Oct 30 19:03:31.825870 ip-10-0-130-217 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:03:32.193572 ip-10-0-130-217 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:03:32.193572 ip-10-0-130-217 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:03:32.193572 ip-10-0-130-217 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:03:32.193572 ip-10-0-130-217 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:03:32.199540 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199463    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.199540 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199536    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.199540 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199542    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199546    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199551    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199555    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199560    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199565    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199568    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199572    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199577    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199581    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199585    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199590    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199610    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199615    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199620    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199624    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199628    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199633    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199637    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199643    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.199757 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199647    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199652    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199656    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199660    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199665    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199669    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199674    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199678    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199683    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199688    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199693    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199698    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199703    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199707    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199712    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199718    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199722    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199727    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199732    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199736    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.200580 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199741    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199746    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199750    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199755    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199759    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199764    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199771    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199779    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199785    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199790    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199795    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199800    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199805    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199809    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199814    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199819    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199823    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199828    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199833    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199840    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.201443 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199844    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199850    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199854    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199858    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199863    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199868    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199873    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199879    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199883    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199889    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199893    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199898    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199902    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199907    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199912    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199916    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199921    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199928    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199936    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.202317 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199941    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199947    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199953    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199959    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199963    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199968    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199973    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199977    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199982    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199986    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199991    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.199996    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200001    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200006    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200010    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200016    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200021    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200026    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200031    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.203148 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200036    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200041    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200046    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200051    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200055    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.200060    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201363    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201375    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201382    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201388    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201393    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201398    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201403    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201409    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201415    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201420    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201426    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201430    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201435    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201439    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.203766 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201444    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201448    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201453    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201457    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201461    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201466    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201471    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201476    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201480    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201485    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201489    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201494    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201501    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201506    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201511    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201515    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201520    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201525    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201529    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201534    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.204322 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201538    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201543    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201547    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201552    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201562    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201567    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201572    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201577    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201582    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201587    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201592    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201616    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201621    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201625    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201629    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201634    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201640    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201645    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.205218 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201649    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201654    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201658    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201663    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201667    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201674    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201678    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201685    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201691    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201696    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201701    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201706    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201710    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201715    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201719    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201724    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201729    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201734    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201739    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.206094 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201743    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201747    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201752    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201756    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201762    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201767    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201771    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201776    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201780    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201785    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201789    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201794    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201798    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201803    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201808    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201813    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201818    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201822    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201828    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.206805 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201833    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201839    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201843    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201848    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201853    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201858    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201863    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201868    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201872    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201876    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201881    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201886    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201891    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201896    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.201904    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202730    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202742    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202750    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202755    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.207261 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202760    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202764    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202769    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202774    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202778    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202783    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202787    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202792    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202797    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202802    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202807    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202811    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202818    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202823    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202829    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202833    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202838    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202842    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202847    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202851    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.208016 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202855    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202860    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202864    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202869    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202873    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202878    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202883    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202887    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202891    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202896    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202900    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202905    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202910    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202915    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202919    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202924    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202929    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202933    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202937    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.208510 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202942    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202946    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202951    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202955    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202960    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202966    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202971    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202976    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202981    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202985    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202990    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202994    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.202999    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203004    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203010    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203016    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203022    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203027    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203032    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.209101 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203036    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203041    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203046    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203051    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203055    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203060    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203065    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203069    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203074    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203078    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203083    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203088    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203093    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203097    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203102    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203106    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203110    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203115    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203121    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.209550 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203126    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203130    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203136    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203140    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203146    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203153    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203158    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203164    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203169    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203174    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203180    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203185    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203191    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203196    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203201    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203207    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203211    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203216    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203220    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.210077 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203225    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203231    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203235    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203240    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203244    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.203249    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204178    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204193    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204206    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204214    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204221    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204228    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204235    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204242    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204249    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204254    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204260    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204266    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204272    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204277    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204281    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204287    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204293    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204298    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:03:32.210531 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204304    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204309    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204314    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204320    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204326    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204332    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204337    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204342    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204348    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204354    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204360    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204366    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204371    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204376    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204384    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204389    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204394    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204399    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204412    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204417    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204428    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204434    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204439    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204444    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:03:32.211108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204450    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204458    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204463    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204468    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204474    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204479    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204484    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204489    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204495    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204500    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204505    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204510    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204517    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204523    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204528    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204533    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204539    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204545    2581 flags.go:64] FLAG: --help="false"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204550    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204557    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204562    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204568    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204573    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:03:32.211776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204580    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204585    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204590    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204609    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204615    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204623    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204629    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204634    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204640    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204645    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204651    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204656    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204661    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204666    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204671    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204677    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204686    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204692    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204697    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204703    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204708    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204714    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204719    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204724    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204732    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:03:32.212327 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204737    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204744    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204749    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204755    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204761    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204767    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204772    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204777    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204783    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204795    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204800    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204805    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204813    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204818    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204823    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204829    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204835    2581 flags.go:64] FLAG: --port="10250"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204840    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204845    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ba14033210faf6ad"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204850    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204856    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204861    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204867    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204881    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204886    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204891    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:03:32.212946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204896    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204903    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204908    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204914    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204919    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204924    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204929    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204935    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204940    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204945    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204951    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204958    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204964    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204969    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204974    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204980    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204985    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.204990    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205000    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205005    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205010    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205020    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205025    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205030    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205037    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:03:32.213662 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205042    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205047    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205053    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205058    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205064    2581 flags.go:64] FLAG: --v="2"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205072    2581 flags.go:64] FLAG: --version="false"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205078    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205085    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205091    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205302    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205311    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205317    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205322    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205328    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205333    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205338    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205344    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205349    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205354    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205359    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205364    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.214248 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205369    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205373    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205378    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205383    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205391    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205396    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205401    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205406    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205410    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205415    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205419    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205424    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205429    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205434    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205439    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205443    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205448    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205452    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205457    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.214770 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205467    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205472    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205476    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205481    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205485    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205490    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205494    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205499    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205504    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205509    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205514    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205520    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205525    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205530    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205534    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205538    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205547    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205551    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205556    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.215208 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205561    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205565    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205570    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205575    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205579    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205584    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205589    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205609    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205617    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205623    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205628    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205633    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205637    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205642    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205647    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205652    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205656    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205661    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205666    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205671    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.215693 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205675    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205680    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205685    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205689    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205694    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205700    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205705    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205710    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205714    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205722    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205727    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205731    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205736    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205741    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205746    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205751    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205755    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205760    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205765    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.216157 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205770    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205774    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205779    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205784    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205788    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205793    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205798    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205802    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205806    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205811    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205816    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205821    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205825    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205830    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205835    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205840    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.216628 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.205882    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205940    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205948    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205954    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205959    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205964    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205971    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205979    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205984    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205989    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205994    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.205999    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206003    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206007    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206012    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206017    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206022    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206027    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206032    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.217014 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206036    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206043    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206049    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206054    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206059    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206064    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206070    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206075    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206081    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206086    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206090    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206095    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206099    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206104    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206108    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206112    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206117    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206123    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206128    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.217456 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206135    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206140    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206145    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206149    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206154    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206158    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206163    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206168    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206173    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206178    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206182    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206187    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206191    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206196    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206201    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206205    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206209    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206214    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206219    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.217927 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206223    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206228    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206233    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206237    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206243    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206248    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206253    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206257    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206262    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206266    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206271    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206276    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206282    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206288    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206293    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206298    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206303    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206308    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206312    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206317    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.218363 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206321    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206326    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206332    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206338    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206344    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206349    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206353    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206358    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206363    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206367    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206371    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206376    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206381    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206386    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206390    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206395    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206400    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206404    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206409    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.218871 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206413    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206419    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206424    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206429    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206433    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206438    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206445    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206450    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206454    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206459    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.206464    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.213000    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.213015    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.213559    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.213565    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213634    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213640    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213643    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213646    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.219309 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213648    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213651    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213654    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213656    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213659    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213661    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213664    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213667    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213670    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213673    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213675    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213678    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213681    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213683    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213686    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213689    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213691    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213694    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213697    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213699    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.219787 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213702    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213704    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213707    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213709    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213712    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213717    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213719    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213722    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213725    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213727    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213730    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213733    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213735    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213738    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213740    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213743    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213746    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213748    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213752    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.220277 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213756    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213758    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213762    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213765    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213768    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213770    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213773    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213775    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213778    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213781    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213783    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213786    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213788    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213791    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213793    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213796    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213798    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213801    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213804    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213810    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.220744 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213814    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213816    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213819    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213821    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213824    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213827    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213829    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213832    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213834    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213837    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213840    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213842    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213845    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213847    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213850    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213852    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213855    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213858    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213860    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.221335 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213863    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213865    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213868    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213870    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213873    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213876    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213878    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213880    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213883    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213886    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213888    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213891    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213893    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213897    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213899    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213902    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213904    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213908    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213912    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.222057 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213915    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213918    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213920    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213923    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.213925    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.213953    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214081    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214086    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214089    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214091    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214094    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214097    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214099    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214102    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214105    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214108    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.222586 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214111    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214113    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214116    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214118    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214121    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214123    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214126    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214128    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214131    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214134    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214136    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214139    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214142    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214144    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214147    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214149    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214152    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214154    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214157    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214159    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.223026 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214162    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214164    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214167    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214169    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214172    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214174    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214177    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214179    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214182    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214184    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214187    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214189    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214192    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214195    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214197    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214200    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214202    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214205    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214207    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214210    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.223508 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214213    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214215    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214218    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214220    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214224    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214227    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214229    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214232    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214234    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214237    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214239    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214242    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214244    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214247    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214249    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214252    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214255    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214258    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214260    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.224041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214263    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214266    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214268    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214271    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214273    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214276    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214279    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214281    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214284    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214286    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214288    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214291    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214294    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214296    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214299    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214301    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214304    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214307    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214309    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.224741 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214312    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214315    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214317    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214319    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214322    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214325    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214328    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214332    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214334    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214337    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214339    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214342    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214344    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214347    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214349    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214355    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214358    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.225186 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214362    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.214385    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214415    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214419    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214421    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214424    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214427    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214429    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214432    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214435    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214437    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214440    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214443    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214445    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214447    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214450    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:03:32.225608 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214453    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214457    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214460    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214464    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214467    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214469    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214472    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214474    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214477    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214479    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214482    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214484    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214488    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214492    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214495    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214499    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214501    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214504    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214507    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:03:32.226051 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214509    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214512    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214514    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214517    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214520    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214522    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214527    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214529    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214532    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214534    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214537    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214540    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214543    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214546    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214549    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214552    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214554    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214557    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214559    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214562    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:03:32.226526 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214564    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214567    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214569    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214572    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214575    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214577    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214580    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214582    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214586    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214589    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214591    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214605    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214608    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214611    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214614    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214616    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214619    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214621    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214625    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214628    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:03:32.227065 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214631    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214633    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214636    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214638    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214640    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214643    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214646    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214649    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214651    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214655    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214657    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214660    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214662    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214665    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214667    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214670    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214672    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214675    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214677    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214680    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:03:32.227555 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214683    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214686    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214689    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214691    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214694    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214696    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214699    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214701    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214704    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214706    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214710    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214713    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:32.214715    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.214764    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.217965    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.219395    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.219486    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.219525    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:03:32.228041 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.221040    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:03:32.228503 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.225607    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:03:32.236968 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.236953    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:03:32.240059 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.240045    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 19:03:32.240862 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.240840    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:03:32.243126 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.243100    2581 fs.go:135] Filesystem UUIDs: map[12109c07-cbd9-4dde-af9c-ce22653c9b99:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d6654752-17f9-43db-9dcd-86ac1f1396bb:/dev/nvme0n1p3]
Oct 30 19:03:32.243168 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.243125    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:03:32.247929 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.247824    2581 manager.go:217] Machine: {Timestamp:2026-10-30 19:03:32.246631379 +0000 UTC m=+0.312296437 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105571 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268791b75f0d4fdfe6e0a5a85096fa SystemUUID:ec268791-b75f-0d4f-dfe6-e0a5a85096fa BootID:0a73bff7-c4e9-48dd-b8fe-5440ed31a46f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:c2:c3:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:c2:c3:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:c5:bf:93:17:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:03:32.247929 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.247926    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:03:32.248042 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.247992    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:03:32.250677 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.250653    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:03:32.250805 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.250679    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:03:32.251552 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.251543    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:03:32.251586 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.251554    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:03:32.251586 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.251579    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:03:32.251653 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.251591    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:03:32.252347 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.252337    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:03:32.256302 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.256290    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:03:32.256420 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.256411    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:03:32.257061 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257050    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:03:32.257092 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257067    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:03:32.257092 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257079    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:03:32.257092 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257089    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:03:32.257190 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257097    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:03:32.257403 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.257390    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:03:32.259356 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.259328    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:03:32.265949 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.265933    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:03:32.266741 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266729    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:03:32.266780 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266758    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:03:32.266903 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266892    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:03:32.266942 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266907    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:03:32.266942 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266913    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:03:32.266942 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266918    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:03:32.266942 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266923    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:03:32.266942 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.266929    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:03:32.267542 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267534    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:03:32.267576 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267543    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:03:32.267576 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267551    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:03:32.267576 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267556    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:03:32.267576 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267564    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:03:32.267576 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.267573    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:03:32.268651 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.268641    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:03:32.268651 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.268651    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:03:32.272059 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.272044    2581 server.go:1263] "Started kubelet"
Oct 30 19:03:32.272145 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.272113    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:03:32.272183 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.272130    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:03:32.272211 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.272196    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:03:32.272734 ip-10-0-130-217 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:03:32.273273 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.273255    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:03:32.274025 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.274008    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:03:32.278413 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.277942    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:03:32.281515 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.281498    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:03:32.282410 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.282396    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:03:32.282510 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.282490    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.283036 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283021    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:03:32.283036 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283038    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:03:32.283169 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283022    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:03:32.283169 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283142    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:03:32.283169 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283149    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:03:32.283641 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283628    2581 factory.go:55] Registering systemd factory
Oct 30 19:03:32.283709 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283650    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:03:32.283828 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283817    2581 factory.go:153] Registering CRI-O factory
Oct 30 19:03:32.283905 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283830    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:03:32.283905 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283877    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:03:32.283905 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283889    2581 factory.go:103] Registering Raw factory
Oct 30 19:03:32.283905 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.283899    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:03:32.284721 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.284706    2581 manager.go:319] Starting recovery of all containers
Oct 30 19:03:32.294196 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.294174    2581 manager.go:324] Recovery completed
Oct 30 19:03:32.298931 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.298917    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.300564 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.300537    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.300686 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.300577    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.300686 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.300591    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.301131 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301118    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.301208 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301121    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:03:32.301208 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301147    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:03:32.301208 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301162    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:03:32.301805 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301789    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.301863 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301811    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.301863 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.301821    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.302292 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.302271    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-217.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:03:32.302368 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.302274    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:03:32.303434 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.303415    2581 policy_none.go:50] "Start"
Oct 30 19:03:32.303434 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.303432    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:03:32.303571 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.303448    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:03:32.305484 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.305471    2581 policy_none.go:44] "Start"
Oct 30 19:03:32.308759 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.308739    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:03:32.308853 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.308828    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49b8q"
Oct 30 19:03:32.308853 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.308844    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49b8q"
Oct 30 19:03:32.313626 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.312700    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-217.ec2.internal.18e363fc6345d98f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-217.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-217.ec2.internal,},FirstTimestamp:2026-10-30 19:03:32.272019855 +0000 UTC m=+0.337684913,LastTimestamp:2026-10-30 19:03:32.272019855 +0000 UTC m=+0.337684913,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-217.ec2.internal,}"
Oct 30 19:03:32.323814 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.323786    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:03:32.323891 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.323822    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:03:32.334185 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.334153    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:03:32.339191 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.339177    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:03:32.339727 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.339712    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:03:32.339727 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.339728    2581 server.go:87] "Starting device plugin registration server"
Oct 30 19:03:32.339928 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.339917    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:03:32.339967 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.339930    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:03:32.340074 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.340058    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:03:32.340156 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.340127    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:03:32.340156 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.340135    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:03:32.340549 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.340532    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:03:32.340639 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.340575    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.369610 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.369575    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:03:32.410117 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.410094    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:03:32.411208 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.411184    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:03:32.411208 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.411206    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:03:32.411324 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.411226    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:03:32.411324 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.411255    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:03:32.425061 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.425044    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:03:32.440265 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.440253    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.440997 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.440981    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.441073 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.441012    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.441073 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.441023    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.441073 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.441042    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.458192 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.458157    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.458192 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.458176    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-217.ec2.internal\": node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.499992 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.499870    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.512064 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.512042    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"]
Oct 30 19:03:32.512134 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.512107    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.513673 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.513656    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.513748 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.513684    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.513748 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.513695    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.515867 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.515855    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.516014 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.516053 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516030    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.516590 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516573    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.516680 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516616    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.516680 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516632    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.516765 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516633    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.516800 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516765    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.516800 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.516778    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.518658 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.518644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.518736 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.518666    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:32.519431 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.519413    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:32.519503 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.519442    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:32.519503 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.519458    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:32.546135 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.546121    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-217.ec2.internal\" not found" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.548434 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.548419    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-217.ec2.internal\" not found" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.600355 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.600336    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.684709 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.684690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.684773 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.684713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.684773 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.684728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.700689 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.700670    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.785649 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.785649 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.785747 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.785747 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785676    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.785747 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785704    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.785747 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.785708    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.801729 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.801709    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:32.851895 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.851877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.851971 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:32.851908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:32.902456 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:32.902439    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.002939 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.002913    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.103501 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.103449    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.203962 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.203932    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.219100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.219081    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:03:33.219277 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.219251    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:03:33.305086 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.305061    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.313493 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.313466    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:33 +0000 UTC" deadline="2028-06-19 16:09:17.584412239 +0000 UTC"
Oct 30 19:03:33.313493 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.313491    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14349h5m44.270924157s"
Oct 30 19:03:33.405732 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.405711    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.460688 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.460652    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:03:33.463597 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.463575    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:03:33.463719 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.463615    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:03:33.463719 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.463630    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:03:33.506200 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.506176    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.545907 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.545876    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:03:33.572181 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:33.572150    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21287f518f1b139efdbea74cc5c91860.slice/crio-884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99 WatchSource:0}: Error finding container 884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99: Status 404 returned error can't find the container with id 884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99
Oct 30 19:03:33.579397 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.579382    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:03:33.582316 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:33.582290    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32feda196b1df490b5d951791a17beab.slice/crio-a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4 WatchSource:0}: Error finding container a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4: Status 404 returned error can't find the container with id a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4
Oct 30 19:03:33.607318 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.607286    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.707845 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:33.707799    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:03:33.731379 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.731360    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:03:33.784515 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.784492    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:33.812717 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.812700    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:03:33.813757 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.813743    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:03:33.838496 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:33.838481    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:03:34.080176 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.080116    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:03:34.257812 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.257784    2581 apiserver.go:52] "Watching apiserver"
Oct 30 19:03:34.282546 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.282522    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:03:34.306410 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.306386    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:03:34.306771 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.306750    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kgjxh","kube-system/global-pull-secret-syncer-kbslw","kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hxrgs","openshift-image-registry/node-ca-6wmz8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal","openshift-multus/multus-additional-cni-plugins-gq2v8","openshift-multus/network-metrics-daemon-kn5zp","openshift-network-diagnostics/network-check-target-r5zck","kube-system/konnectivity-agent-rrwhk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb","openshift-dns/node-resolver-ckfxz","openshift-multus/multus-nppsv","openshift-network-operator/iptables-alerter-lxwvs"]
Oct 30 19:03:34.309367 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.309339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.312049 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.311749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.314801 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.314781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.316431 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.316412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.320549 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.320473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.322958 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.322938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.323059 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.323010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:34.325961 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.325936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:34.326065 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.326009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:34.327170 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327152    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:03:34.327270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327246    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.327329 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327274    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:03:34.327329 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327320    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.327564 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327544    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:03:34.327705 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.327687    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\""
Oct 30 19:03:34.328894 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.328798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.333123 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.333076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.333204 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.333163    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:34.335471 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.335452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.338737 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.338714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341027    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341165    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341208    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341253    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341293    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341348    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341362    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341388    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341436    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341447    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341453    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341467    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341506    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.341574 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341525    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\""
Oct 30 19:03:34.342370 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.341898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.382416 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.382396    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bc8h7"
Oct 30 19:03:34.384632 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.384556    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:03:34.396419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.396537 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.396537 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.396537 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvx8\" (UniqueName: \"kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.396537 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzdvb\" (UniqueName: \"kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.396790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.396990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7v7\" (UniqueName: \"kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.397081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.397445 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jnvt\" (UniqueName: \"kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp6x\" (UniqueName: \"kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.398858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.397988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhbd\" (UniqueName: \"kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.399352 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2pr\" (UniqueName: \"kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w6f\" (UniqueName: \"kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kx7m\" (UniqueName: \"kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.399994 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.400777 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.400777 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.398822    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bc8h7"
Oct 30 19:03:34.418684 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.418615    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" event={"ID":"32feda196b1df490b5d951791a17beab","Type":"ContainerStarted","Data":"a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4"}
Oct 30 19:03:34.419993 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.419969    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerStarted","Data":"884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99"}
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.435573    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.435629    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.435748    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.435864    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.435971    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436016    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436013    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436164    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436364    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:03:34.436419 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436385    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:03:34.436966 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436552    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\""
Oct 30 19:03:34.436966 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.436693    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:03:34.443825 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.443803    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:03:34.443940 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.443803    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\""
Oct 30 19:03:34.444062 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.444037    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\""
Oct 30 19:03:34.444207 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.444182    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:03:34.444383 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.444360    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\""
Oct 30 19:03:34.499758 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdvb\" (UniqueName: \"kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.499758 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.499959 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:34.499959 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.499959 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.499959 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.499959 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.499993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500038    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.500236 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.500674 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500447    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:03:34.500674 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500534    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.500788 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500719    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.500850 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.500850 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500805    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.500850 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501003 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500869    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501003 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501003 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7v7\" (UniqueName: \"kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.501003 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.501003 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.500988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501073    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501270 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnvt\" (UniqueName: \"kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp6x\" (UniqueName: \"kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501610    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.501869 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501733    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501742    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501777    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501818    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.501831    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502255    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502284    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502331    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502349    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502382    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.502809 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502436    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.502463    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502479    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502495    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.502556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.002515322 +0000 UTC m=+3.068180390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502584    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.502726    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.502783    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.002768792 +0000 UTC m=+3.068433860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502785    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.503666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502857    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhbd\" (UniqueName: \"kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502935    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502957    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.502979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503015    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503059    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503065    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503107    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.504766 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2pr\" (UniqueName: \"kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503200    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503202    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w6f\" (UniqueName: \"kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503278    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503339    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kx7m\" (UniqueName: \"kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503415    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503470    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.505703 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503561    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503571    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503589    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503622    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503669    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503673    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503719    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503750    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503758    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.506384 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503866    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvx8\" (UniqueName: \"kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.503975    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.507100 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.504013    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.507859 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.507669    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.507859 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.507747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.507859 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.507800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.508023 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.507906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.520222 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.520202    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:34.520222 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.520223    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:34.520393 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.520235    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:34.520393 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:34.520291    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.020276088 +0000 UTC m=+3.085941166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:34.522872 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.522845    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdvb\" (UniqueName: \"kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.530188 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.529809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp6x\" (UniqueName: \"kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.531829 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.531803    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7v7\" (UniqueName: \"kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.536326 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.536304    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kx7m\" (UniqueName: \"kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.536429 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.536332    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnvt\" (UniqueName: \"kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.536429 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.536369    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w6f\" (UniqueName: \"kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:34.536909 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.536864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2pr\" (UniqueName: \"kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.537028 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.537006    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhbd\" (UniqueName: \"kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.538598 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.538570    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvx8\" (UniqueName: \"kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.643297 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.643062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:03:34.646001 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.645965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nppsv"
Oct 30 19:03:34.649919 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.649888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:03:34.652189 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.652141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:03:34.654457 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.654430    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0885c92_5132_4361_901c_4530f23a5c57.slice/crio-e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187 WatchSource:0}: Error finding container e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187: Status 404 returned error can't find the container with id e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187
Oct 30 19:03:34.660078 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.660054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:03:34.660345 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.660319    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b52d98_fb0b_4145_aaf5_dc3301172e4b.slice/crio-c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663 WatchSource:0}: Error finding container c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663: Status 404 returned error can't find the container with id c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663
Oct 30 19:03:34.665905 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.665848    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c416fac_c47f_487a_94d8_d27d779ad0d5.slice/crio-8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d WatchSource:0}: Error finding container 8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d: Status 404 returned error can't find the container with id 8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d
Oct 30 19:03:34.672279 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.672254    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c65674_cd6d_45e7_a7ad_2469ca4910cb.slice/crio-206acab0573fabfa54796122b56b4fafdd1f284a7602469bd9b775ae81b2bfa9 WatchSource:0}: Error finding container 206acab0573fabfa54796122b56b4fafdd1f284a7602469bd9b775ae81b2bfa9: Status 404 returned error can't find the container with id 206acab0573fabfa54796122b56b4fafdd1f284a7602469bd9b775ae81b2bfa9
Oct 30 19:03:34.672772 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.672715    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56513e8_bfbe_4662_a549_6aaf87a53316.slice/crio-638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854 WatchSource:0}: Error finding container 638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854: Status 404 returned error can't find the container with id 638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854
Oct 30 19:03:34.677247 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.677225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:34.686860 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.686837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:34.688370 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.688346    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc31816_a783_44bd_83e4_a009e13da8b4.slice/crio-9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551 WatchSource:0}: Error finding container 9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551: Status 404 returned error can't find the container with id 9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551
Oct 30 19:03:34.694566 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.694546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:03:34.697358 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:34.697331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:03:34.703634 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.703559    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeb84c0_12d1_4c7b_9001_42a3f6b1391a.slice/crio-393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316 WatchSource:0}: Error finding container 393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316: Status 404 returned error can't find the container with id 393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316
Oct 30 19:03:34.708202 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.708173    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aba8d2a_7b5b_4265_b8ca_48cac068159d.slice/crio-f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c WatchSource:0}: Error finding container f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c: Status 404 returned error can't find the container with id f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c
Oct 30 19:03:34.710997 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:03:34.710891    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2357170c_bfb3_4ee9_925f_e84d351a2c86.slice/crio-ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff WatchSource:0}: Error finding container ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff: Status 404 returned error can't find the container with id ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff
Oct 30 19:03:35.008838 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.008747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:35.008990 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.008840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:35.008990 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.008976    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:35.009113 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.009038    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:36.009019047 +0000 UTC m=+4.074684096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:35.009432 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.009415    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:35.009478 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.009468    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:36.009452594 +0000 UTC m=+4.075117646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:35.109126 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.109096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:35.109245 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.109227    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:35.109306 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.109252    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:35.109306 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.109266    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:35.109306 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.109305    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:36.10929334 +0000 UTC m=+4.174958402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:35.402369 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.402024    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:35 +0000 UTC" deadline="2028-06-05 07:31:08.539020966 +0000 UTC"
Oct 30 19:03:35.402369 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.402307    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14004h27m33.136720218s"
Oct 30 19:03:35.412606 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.412109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:35.412606 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:35.412258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:35.435237 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.434990    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" event={"ID":"32feda196b1df490b5d951791a17beab","Type":"ContainerStarted","Data":"82638b06bc82a1dc45bae9c6af28814d9985cc902c21cd55d0c5694522e419fa"}
Oct 30 19:03:35.443473 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.443383    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wmz8" event={"ID":"b56513e8-bfbe-4662-a549-6aaf87a53316","Type":"ContainerStarted","Data":"638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854"}
Oct 30 19:03:35.445545 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.445516    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lxwvs" event={"ID":"4c416fac-c47f-487a-94d8-d27d779ad0d5","Type":"ContainerStarted","Data":"8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d"}
Oct 30 19:03:35.460444 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.460417    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ckfxz" event={"ID":"2357170c-bfb3-4ee9-925f-e84d351a2c86","Type":"ContainerStarted","Data":"ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff"}
Oct 30 19:03:35.475790 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.475760    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb" event={"ID":"7aba8d2a-7b5b-4265-b8ca-48cac068159d","Type":"ContainerStarted","Data":"f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c"}
Oct 30 19:03:35.492654 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.492411    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316"}
Oct 30 19:03:35.497178 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.497097    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rrwhk" event={"ID":"cbc31816-a783-44bd-83e4-a009e13da8b4","Type":"ContainerStarted","Data":"9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551"}
Oct 30 19:03:35.503252 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.503148    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxrgs" event={"ID":"49c65674-cd6d-45e7-a7ad-2469ca4910cb","Type":"ContainerStarted","Data":"206acab0573fabfa54796122b56b4fafdd1f284a7602469bd9b775ae81b2bfa9"}
Oct 30 19:03:35.506253 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.506171    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nppsv" event={"ID":"c4b52d98-fb0b-4145-aaf5-dc3301172e4b","Type":"ContainerStarted","Data":"c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663"}
Oct 30 19:03:35.512627 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:35.512533    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerStarted","Data":"e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187"}
Oct 30 19:03:36.016005 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.015967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:36.016203 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.016064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:36.016272 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.016243    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:36.016335 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.016308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:38.016287693 +0000 UTC m=+6.081952743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:36.016758 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.016739    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:36.016846 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.016801    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:38.016783946 +0000 UTC m=+6.082449020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:36.116747 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.116670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:36.116891 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.116879    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:36.116963 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.116902    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:36.116963 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.116916    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:36.117062 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.116980    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:38.116964106 +0000 UTC m=+6.182629159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:36.403632 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.403572    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:35 +0000 UTC" deadline="2028-05-11 10:36:49.384540134 +0000 UTC"
Oct 30 19:03:36.403632 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.403630    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13407h33m12.980914659s"
Oct 30 19:03:36.412778 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.412746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:36.412778 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.412765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:36.412952 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.412897    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:36.413282 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:36.413044    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:36.528481 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.528443    2581 generic.go:356] "Generic (PLEG): container finished" podID="21287f518f1b139efdbea74cc5c91860" containerID="b18f263eceb2052f3f3d194ec4411496f763a2f7e4fb82e73198c1338c9bc877" exitCode=0
Oct 30 19:03:36.529247 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.528726    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerDied","Data":"b18f263eceb2052f3f3d194ec4411496f763a2f7e4fb82e73198c1338c9bc877"}
Oct 30 19:03:36.572420 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:36.572355    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" podStartSLOduration=2.572336986 podStartE2EDuration="2.572336986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:35.480820294 +0000 UTC m=+3.546485365" watchObservedRunningTime="2026-10-30 19:03:36.572336986 +0000 UTC m=+4.638002058"
Oct 30 19:03:37.411533 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:37.411505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:37.411983 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:37.411609    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:37.545261 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:37.545224    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="64cb33f98c0108cd8f577a3eef5d161d983c37814a8231b30747ed3161a31487" exitCode=0
Oct 30 19:03:37.545423 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:37.545366    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"64cb33f98c0108cd8f577a3eef5d161d983c37814a8231b30747ed3161a31487"}
Oct 30 19:03:37.559267 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:37.559239    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerStarted","Data":"e42e3855ec68ed131fe72f585dd7606437cf97e6178cca90c8d4d67e5112dccf"}
Oct 30 19:03:38.036190 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:38.036154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:38.036391 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:38.036217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:38.036391 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.036342    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:38.036505 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.036402    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:42.036382587 +0000 UTC m=+10.102047650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:38.037060 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.036772    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:38.037060 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.036836    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:42.036818689 +0000 UTC m=+10.102483755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:38.136645 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:38.136595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:38.136836 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.136812    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:38.136927 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.136843    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:38.136927 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.136856    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:38.136927 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.136917    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:42.136898425 +0000 UTC m=+10.202563486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:38.414054 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:38.414022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:38.414532 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.414153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:38.414597 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:38.414578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:38.414900 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:38.414873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:39.411689 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:39.411642    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:39.411905 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:39.411777    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:39.572793 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:39.572052    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rrwhk" event={"ID":"cbc31816-a783-44bd-83e4-a009e13da8b4","Type":"ContainerStarted","Data":"be440dff9ad9e7681c788a361e37d780b6292b96a7fb58941244b50bf39ae4ec"}
Oct 30 19:03:39.618067 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:39.616720    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" podStartSLOduration=5.6166994070000005 podStartE2EDuration="5.616699407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:37.638474251 +0000 UTC m=+5.704139323" watchObservedRunningTime="2026-10-30 19:03:39.616699407 +0000 UTC m=+7.682364480"
Oct 30 19:03:40.412779 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:40.412746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:40.412973 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:40.412879    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:40.413308 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:40.413288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:40.413425 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:40.413402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:40.585580 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:40.585542    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wmz8" event={"ID":"b56513e8-bfbe-4662-a549-6aaf87a53316","Type":"ContainerStarted","Data":"86d8983c50b9dcac424256be37a1d2c0fb7bd33fd6f9ed949ee2c4820174faf1"}
Oct 30 19:03:40.630193 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:40.630131    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rrwhk" podStartSLOduration=3.807218716 podStartE2EDuration="7.630108701s" totalImagesPullingTime="3.822889985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:39.618050325 +0000 UTC m=+7.683715396" watchObservedRunningTime="2026-10-30 19:03:40.630108701 +0000 UTC m=+8.695773769"
Oct 30 19:03:41.411458 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:41.411423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:41.411653 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:41.411574    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:41.591013 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:41.590973    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb" event={"ID":"7aba8d2a-7b5b-4265-b8ca-48cac068159d","Type":"ContainerStarted","Data":"630102cbe6ec4eb1ef7c972e8f8839c38a18c289314ae710979ca94ff918f850"}
Oct 30 19:03:41.830946 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:41.830344    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:41.831120 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:41.831094    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:41.876984 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:41.876924    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6wmz8" podStartSLOduration=3.464115275 podStartE2EDuration="8.876904402s" totalImagesPullingTime="5.412789127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:40.631582516 +0000 UTC m=+8.697247604" watchObservedRunningTime="2026-10-30 19:03:41.876904402 +0000 UTC m=+9.942569475"
Oct 30 19:03:42.075315 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.075284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:42.075469 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.075346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:42.075469 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.075440    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:42.075537 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.075485    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.075472866 +0000 UTC m=+18.141137912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:42.075537 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.075527    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:42.075605 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.075549    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.075541163 +0000 UTC m=+18.141206210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:42.176106 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.176066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:42.176299 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.176280    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:42.176352 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.176309    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:42.176352 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.176325    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:42.176420 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.176392    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:50.176373233 +0000 UTC m=+18.242038305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:42.412769 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.412733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:42.412941 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.412854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:42.413341 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.413311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:42.413459 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:42.413414    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:42.594911 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.594849    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:42.595596 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.595539    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:03:42.638235 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:42.638195    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:03:43.346949 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.346721    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:03:42.638217171Z","UUID":"88138203-f85b-410b-b8e5-b5df282b1a35","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:03:43.348339 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.348318    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:03:43.348442 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.348350    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:03:43.411955 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.411894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:43.412079 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:43.411986    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:43.598216 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.598178    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb" event={"ID":"7aba8d2a-7b5b-4265-b8ca-48cac068159d","Type":"ContainerStarted","Data":"28b480c7186bf9dfcece84c6b2daf36a44b2c0c8c2ff78f89397712b87f52ecc"}
Oct 30 19:03:43.600222 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.600178    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nppsv" event={"ID":"c4b52d98-fb0b-4145-aaf5-dc3301172e4b","Type":"ContainerStarted","Data":"62480ba80bc10fefdefe4dfc1f9482e107492d88959cce70c9d180d35d221d1c"}
Oct 30 19:03:43.602316 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.602281    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="5437eb0edad2ee97a44c7408020005159613f9dfbbd162246e1aa135d598d582" exitCode=0
Oct 30 19:03:43.602412 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.602360    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"5437eb0edad2ee97a44c7408020005159613f9dfbbd162246e1aa135d598d582"}
Oct 30 19:03:43.604184 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.604161    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ckfxz" event={"ID":"2357170c-bfb3-4ee9-925f-e84d351a2c86","Type":"ContainerStarted","Data":"a07e4e85a79bbc86b267fce59320a17d64a53b45fe981c580479a95ba31eb969"}
Oct 30 19:03:43.644858 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.644824    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nppsv" podStartSLOduration=2.403843757 podStartE2EDuration="10.644812255s" totalImagesPullingTime="8.240968498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:43.644192336 +0000 UTC m=+11.709857404" watchObservedRunningTime="2026-10-30 19:03:43.644812255 +0000 UTC m=+11.710477318"
Oct 30 19:03:43.731586 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:43.731072    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ckfxz" podStartSLOduration=2.298805797 podStartE2EDuration="10.731057597s" totalImagesPullingTime="8.4322518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:43.731044542 +0000 UTC m=+11.796709611" watchObservedRunningTime="2026-10-30 19:03:43.731057597 +0000 UTC m=+11.796722666"
Oct 30 19:03:44.414314 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.414290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:44.414428 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:44.414406    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:44.414505 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.414491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:44.414625 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:44.414587    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:44.608532 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.608504    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="a68655a4d89a39c2db1e29975ec6d2a5576f1bfd4aa8c604e3176da649ae3bcc" exitCode=0
Oct 30 19:03:44.609045 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.608607    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"a68655a4d89a39c2db1e29975ec6d2a5576f1bfd4aa8c604e3176da649ae3bcc"}
Oct 30 19:03:44.610414 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.610386    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lxwvs" event={"ID":"4c416fac-c47f-487a-94d8-d27d779ad0d5","Type":"ContainerStarted","Data":"6ff4f41ee7542248ff0207746aa9f27f049def45cae8b556378acf9b04a055bf"}
Oct 30 19:03:44.613034 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.613002    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb" event={"ID":"7aba8d2a-7b5b-4265-b8ca-48cac068159d","Type":"ContainerStarted","Data":"964d2c0bcad8526af32c2a696b022d3dd91550d21c3ec1f2fc60e492601af2c2"}
Oct 30 19:03:44.720241 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.719907    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lxwvs" podStartSLOduration=3.147431661 podStartE2EDuration="11.719888471s" totalImagesPullingTime="8.57245681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.719209291 +0000 UTC m=+12.784874369" watchObservedRunningTime="2026-10-30 19:03:44.719888471 +0000 UTC m=+12.785553539"
Oct 30 19:03:44.761636 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:44.761584    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb" podStartSLOduration=3.200103875 podStartE2EDuration="11.761575278s" totalImagesPullingTime="8.561471403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.761379909 +0000 UTC m=+12.827044978" watchObservedRunningTime="2026-10-30 19:03:44.761575278 +0000 UTC m=+12.827240346"
Oct 30 19:03:45.412312 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:45.412284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:45.412433 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:45.412382    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:45.616639 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:45.616610    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxrgs" event={"ID":"49c65674-cd6d-45e7-a7ad-2469ca4910cb","Type":"ContainerStarted","Data":"2f4075774faddd097393447890c650b16206ad2f4b2bb1433fd8525e70f37354"}
Oct 30 19:03:45.659023 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:45.658971    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hxrgs" podStartSLOduration=2.046404387 podStartE2EDuration="12.658951553s" totalImagesPullingTime="10.612547166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:45.658137175 +0000 UTC m=+13.723802242" watchObservedRunningTime="2026-10-30 19:03:45.658951553 +0000 UTC m=+13.724616621"
Oct 30 19:03:46.411758 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:46.411722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:46.411924 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:46.411758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:46.411924 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:46.411873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:46.412051 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:46.412021    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:46.621776 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:46.621746    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="9c3f2b663fef8bd42711bbf26eaa53a93550c2692e6724a8c9fd36493c0803c0" exitCode=0
Oct 30 19:03:46.622398 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:46.621841    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"9c3f2b663fef8bd42711bbf26eaa53a93550c2692e6724a8c9fd36493c0803c0"}
Oct 30 19:03:47.411950 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:47.411755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:47.412095 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:47.412039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:48.412136 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:48.412110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:48.412524 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:48.412199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:48.412524 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:48.412262    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:48.412637 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:48.412576    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:48.630238 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:48.630154    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="704c3e3533233abde9e6f46a3bb33cda4d80584922049f02e2fd8e9a135809c5" exitCode=0
Oct 30 19:03:48.630238 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:48.630222    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"704c3e3533233abde9e6f46a3bb33cda4d80584922049f02e2fd8e9a135809c5"}
Oct 30 19:03:49.412243 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:49.412216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:49.412628 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:49.412311    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:49.635207 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:49.635177    2581 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="090e60d5e877fdc43a9b225287efd2d72e213ff764e3a44aa9095861856e4db5" exitCode=0
Oct 30 19:03:49.635334 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:49.635229    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerDied","Data":"090e60d5e877fdc43a9b225287efd2d72e213ff764e3a44aa9095861856e4db5"}
Oct 30 19:03:50.131757 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.131722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:50.131880 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.131781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:50.131880 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.131854    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:50.131953 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.131888    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:50.131953 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.131906    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:06.131893283 +0000 UTC m=+34.197558342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:03:50.131953 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.131935    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret podName:0261b7c5-c9bb-4fb6-aa6d-23914249b848 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:06.131920691 +0000 UTC m=+34.197585756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret") pod "global-pull-secret-syncer-kbslw" (UID: "0261b7c5-c9bb-4fb6-aa6d-23914249b848") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:03:50.232973 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.232914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:50.233061 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.233022    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:03:50.233061 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.233040    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:03:50.233061 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.233050    2581 projected.go:196] Error preparing data for projected volume kube-api-access-l59h2 for pod openshift-network-diagnostics/network-check-target-r5zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:50.233157 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.233083    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2 podName:d54891c7-b195-47ad-8bee-911c66922826 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:06.233073714 +0000 UTC m=+34.298738760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l59h2" (UniqueName: "kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2") pod "network-check-target-r5zck" (UID: "d54891c7-b195-47ad-8bee-911c66922826") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:03:50.412278 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.412251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:50.412635 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.412285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:50.412635 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.412359    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:50.412635 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:50.412414    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:50.639853 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.639824    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"84d24ff9f39e7c876529db961da1b349d4979bb0c59db27fb9f7acbb630202cb"}
Oct 30 19:03:50.639973 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.639932    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"dd4ce4b9083e1d13c9f7cee60474a4dc40268f6a4d453e13c0c002136f8eae7c"}
Oct 30 19:03:50.640016 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.639973    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"e04860b6c39e388878840d7e591ea55fd0d5ef4aadfca5a5aba9586ee085525c"}
Oct 30 19:03:50.640016 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.640008    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"3e3438907b5a887d9c718631988a39d1f0bd517731d5f6cce5409cb359a6ca02"}
Oct 30 19:03:50.640077 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.640046    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"ae971e6f8c9fe72e36bb31a9d72f336523def5e3989998de375b07fe2618c980"}
Oct 30 19:03:50.640128 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.640082    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"a2c7fb288f6df0fcbadcfa389a151d154596897dca41037f7836c1c31d6535ad"}
Oct 30 19:03:50.643412 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.643380    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" event={"ID":"c0885c92-5132-4361-901c-4530f23a5c57","Type":"ContainerStarted","Data":"2becfc703626af2439c28e6d6057bc2a7de8f45d34389286f21ee104370ac056"}
Oct 30 19:03:50.689504 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:50.689455    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gq2v8" podStartSLOduration=7.304314049 podStartE2EDuration="17.689443441s" totalImagesPullingTime="10.385129392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:50.689028305 +0000 UTC m=+18.754693389" watchObservedRunningTime="2026-10-30 19:03:50.689443441 +0000 UTC m=+18.755108508"
Oct 30 19:03:51.411865 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:51.411839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:51.411993 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:51.411934    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:52.413098 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:52.413074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:52.413484 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:52.413153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:52.413484 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:52.413246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:52.413484 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:52.413353    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:52.651108 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:52.651079    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"a76b02abb4b958c8a4b84b0172fbe364d134acbef569110da5ba9465a0c335d7"}
Oct 30 19:03:53.411743 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:53.411708    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:53.411887 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:53.411804    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:54.411521 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.411488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:54.412152 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:54.411656    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:54.412152 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.411702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:54.412152 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:54.411794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:54.658165 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.658138    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" event={"ID":"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a","Type":"ContainerStarted","Data":"4909b5c5d1499ce0d2d4f750791086fea646435d3d023e39fbac617cba4f8247"}
Oct 30 19:03:54.658356 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.658337    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:54.659492 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.659455    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:54.675083 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.675061    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:54.675167 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.675149    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:03:54.748267 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:54.748224    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh" podStartSLOduration=6.695095845 podStartE2EDuration="21.748213719s" totalImagesPullingTime="15.053117874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:54.747942265 +0000 UTC m=+22.813607332" watchObservedRunningTime="2026-10-30 19:03:54.748213719 +0000 UTC m=+22.813878786"
Oct 30 19:03:55.411908 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:55.411867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:55.412324 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:55.412007    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:55.659816 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:55.659789    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:03:56.234636 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.234386    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kbslw"]
Oct 30 19:03:56.234636 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.234616    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r5zck"]
Oct 30 19:03:56.234879 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.234690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:56.234879 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:56.234768    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:56.235081 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.235065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:56.235160 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:56.235142    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:56.247343 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.247316    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kn5zp"]
Oct 30 19:03:56.247443 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.247407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:56.247500 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:56.247485    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:03:56.662273 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:56.662250    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:03:58.412331 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:58.412299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:03:58.412753 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:58.412366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:03:58.412753 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:03:58.412363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:03:58.412753 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:58.412472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:03:58.412753 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:58.412572    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:03:58.412753 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:03:58.412705    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:04:00.414930 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:00.414900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:00.415315 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:00.415025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:00.415315 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:00.415049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:00.415315 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:00.415039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:04:00.415315 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:00.415128    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:04:00.415315 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:00.415198    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:04:02.412818 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:02.412792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:02.413200 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:02.412868    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r5zck" podUID="d54891c7-b195-47ad-8bee-911c66922826"
Oct 30 19:04:02.413200 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:02.412967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:02.413200 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:02.413065    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kn5zp" podUID="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7"
Oct 30 19:04:02.413200 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:02.413106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:02.413200 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:02.413154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbslw" podUID="0261b7c5-c9bb-4fb6-aa6d-23914249b848"
Oct 30 19:04:02.749292 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:02.749267    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeReady"
Oct 30 19:04:02.749472 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:02.749397    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:04:04.145487 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.145453    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:04.145892 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.145695    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:04:04.166780 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.166748    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:04.411710 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.411642    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:04.411849 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.411757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:04.412020 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.411981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:04.428410 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428386    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:04:04.428530 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428419    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:04:04.428530 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428525    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:04:04.428659 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428553    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\""
Oct 30 19:04:04.428659 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428562    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\""
Oct 30 19:04:04.428659 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428557    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:04:04.428659 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:04.428591    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\""
Oct 30 19:04:06.149458 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.149431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:06.149832 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.149471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:06.149832 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:06.149567    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:04:06.149832 ip-10-0-130-217 kubenswrapper[2581]: E1030 19:04:06.149647    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs podName:ca3c9c4d-c2a2-4fb7-a250-396d97523ad7 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:38.14963401 +0000 UTC m=+66.215299060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs") pod "network-metrics-daemon-kn5zp" (UID: "ca3c9c4d-c2a2-4fb7-a250-396d97523ad7") : secret "metrics-daemon-secret" not found
Oct 30 19:04:06.153130 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.153109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:06.250291 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.250264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:06.252503 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.252482    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:06.317666 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.317645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:06.330808 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.330792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:06.694198 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.694171    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kbslw"]
Oct 30 19:04:06.699637 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:04:06.699587    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0261b7c5_c9bb_4fb6_aa6d_23914249b848.slice/crio-ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa WatchSource:0}: Error finding container ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa: Status 404 returned error can't find the container with id ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa
Oct 30 19:04:06.707571 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:06.707548    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r5zck"]
Oct 30 19:04:06.710949 ip-10-0-130-217 kubenswrapper[2581]: W1030 19:04:06.710925    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54891c7_b195_47ad_8bee_911c66922826.slice/crio-b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856 WatchSource:0}: Error finding container b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856: Status 404 returned error can't find the container with id b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856
Oct 30 19:04:07.687160 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:07.687118    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kbslw" event={"ID":"0261b7c5-c9bb-4fb6-aa6d-23914249b848","Type":"ContainerStarted","Data":"ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa"}
Oct 30 19:04:07.688780 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:07.688726    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r5zck" event={"ID":"d54891c7-b195-47ad-8bee-911c66922826","Type":"ContainerStarted","Data":"b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856"}
Oct 30 19:04:08.120835 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:08.120771    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ckfxz_2357170c-bfb3-4ee9-925f-e84d351a2c86/dns-node-resolver/0.log"
Oct 30 19:04:09.070312 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:09.070284    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wmz8_b56513e8-bfbe-4662-a549-6aaf87a53316/node-ca/0.log"
Oct 30 19:04:10.698162 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:10.698121    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r5zck" event={"ID":"d54891c7-b195-47ad-8bee-911c66922826","Type":"ContainerStarted","Data":"2f8fbf250327f7285524d13356c6e5f9d8153c240a7cf005b87e95ca10d4df75"}
Oct 30 19:04:10.698612 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:10.698218    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:10.739509 ip-10-0-130-217 kubenswrapper[2581]: I1030 19:04:10.739462    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r5zck" podStartSLOduration=34.40776344 podStartE2EDuration="37.739449468s" totalImagesPullingTime="3.331686028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:10.738851048 +0000 UTC m=+38.804516116" watchObservedRunningTime="2026-10-30 19:04:10.739449468 +0000 UTC m=+38.805114535"
Oct 30 19:04:11.906618 ip-10-0-130-217 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:04:11.915970 ip-10-0-130-217 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:04:11.916137 ip-10-0-130-217 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:04:11.916183 ip-10-0-130-217 systemd[1]: kubelet.service: Consumed 1.990s CPU time.
Oct 30 19:04:11.936025 ip-10-0-130-217 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:04:12.019948 ip-10-0-130-217 kubenswrapper[4320]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:04:12.020507 ip-10-0-130-217 kubenswrapper[4320]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:04:12.020507 ip-10-0-130-217 kubenswrapper[4320]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:04:12.020681 ip-10-0-130-217 kubenswrapper[4320]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:04:12.022853 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022789    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.022853 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022854    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022858    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022862    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022865    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022867    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022870    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022873    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022876    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022879    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022881    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022884    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022887    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022889    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022892    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022895    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022898    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022901    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022903    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022907    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.022932 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022909    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022912    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022915    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022918    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022920    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022923    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022925    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022928    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022931    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022934    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022937    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022940    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022943    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022946    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022948    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022951    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022954    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022956    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022959    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022962    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.023405 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022964    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022967    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022969    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022972    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022974    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022977    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022980    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022982    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022985    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022987    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022990    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022992    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022995    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.022997    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023001    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023004    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023007    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023009    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023011    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023014    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.023924 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023017    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023019    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023022    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023025    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023028    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023031    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023033    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023036    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023038    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023041    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023043    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023046    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023048    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023051    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023053    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023056    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023058    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023061    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023065    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.024466 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023069    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023072    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023076    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023078    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023081    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023084    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023086    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023091    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023093    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023096    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023099    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023101    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023104    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023106    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023109    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023112    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023114    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023118    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023120    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023123    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.024971 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023125    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023128    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023131    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023133    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023136    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023139    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023143    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023932    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023940    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023944    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023946    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023951    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023955    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023958    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023962    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023965    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023968    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023971    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023973    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.025478 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023977    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023980    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023982    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023985    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023987    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023990    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023993    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023995    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.023998    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024000    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024003    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024006    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024008    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024011    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024014    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024017    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024019    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024022    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024024    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024027    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.025958 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024030    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024032    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024035    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024039    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024042    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024046    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024049    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024051    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024054    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024057    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024060    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024063    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024066    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024068    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024071    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024073    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024076    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024078    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024081    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.026454 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024083    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024086    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024088    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024091    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024093    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024096    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024098    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024101    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024103    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024106    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024108    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024111    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024113    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024115    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024118    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024121    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024124    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024126    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024129    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024132    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.026961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024134    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024137    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024140    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024143    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024146    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024148    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024151    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024154    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024156    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024159    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024161    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024164    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024166    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024169    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024171    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024174    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024176    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024179    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024181    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024184    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.027470 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024186    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024189    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024192    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024194    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024197    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024200    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024202    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024205    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024208    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024211    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024213    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024215    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024218    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024220    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024223    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024700    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024706    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024709    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024714    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024718    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.027981 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024722    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024726    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024730    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024734    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024739    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024743    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024747    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024751    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024756    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024760    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024764    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024769    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024773    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024777    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024781    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024786    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024790    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024795    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024799    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024803    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.028507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024808    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024812    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024816    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024820    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024824    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024828    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024830    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024833    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024840    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024843    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024845    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024848    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024850    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024853    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024856    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024858    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024861    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024865    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024869    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.029046 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024872    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024874    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024877    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024879    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024882    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024886    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024889    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024892    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024894    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024897    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024899    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024902    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024904    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024907    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024909    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024912    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024915    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024917    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024920    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.029517 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024922    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024925    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024930    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024934    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024936    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024939    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024941    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024944    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024947    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024949    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024951    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024954    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024956    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024959    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024962    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024964    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024967    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024969    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024975    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.030036 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024977    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024980    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024983    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024986    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024988    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024991    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024993    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024996    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.024999    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025002    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025004    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025007    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025009    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025012    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025014    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025019    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025021    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025028    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025031    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025034    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.030507 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025038    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025041    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025044    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025046    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025140    4320 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025150    4320 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025157    4320 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025162    4320 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025166    4320 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025169    4320 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025173    4320 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025179    4320 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025182    4320 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025185    4320 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025189    4320 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025192    4320 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025195    4320 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025198    4320 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025201    4320 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025204    4320 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025207    4320 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025210    4320 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025213    4320 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:04:12.031016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025216    4320 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025219    4320 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025222    4320 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025225    4320 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025230    4320 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025234    4320 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025237    4320 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025241    4320 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025244    4320 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025247    4320 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025250    4320 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025253    4320 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025256    4320 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025260    4320 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025263    4320 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025266    4320 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025269    4320 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025272    4320 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025275    4320 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025280    4320 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025285    4320 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025288    4320 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025291    4320 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025293    4320 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:04:12.031568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025298    4320 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025301    4320 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025303    4320 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025307    4320 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025310    4320 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025312    4320 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025315    4320 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025318    4320 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025321    4320 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025324    4320 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025327    4320 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025331    4320 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025335    4320 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025338    4320 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025341    4320 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025344    4320 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025347    4320 flags.go:64] FLAG: --help="false"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025350    4320 flags.go:64] FLAG: --hostname-override="ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025353    4320 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025356    4320 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025359    4320 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025363    4320 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025366    4320 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025369    4320 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:04:12.032190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025372    4320 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025375    4320 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025378    4320 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025381    4320 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025386    4320 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025389    4320 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025392    4320 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025394    4320 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025398    4320 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025401    4320 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025404    4320 flags.go:64] FLAG: --lock-file=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025406    4320 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025409    4320 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025412    4320 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025418    4320 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025421    4320 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025424    4320 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025426    4320 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025429    4320 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025433    4320 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025437    4320 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025440    4320 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025445    4320 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025448    4320 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025452    4320 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:04:12.032876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025455    4320 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025459    4320 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025462    4320 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025464    4320 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025468    4320 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025471    4320 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025474    4320 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025481    4320 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025484    4320 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025487    4320 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025490    4320 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025494    4320 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025497    4320 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025500    4320 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025503    4320 flags.go:64] FLAG: --port="10250"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025506    4320 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025508    4320 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ba14033210faf6ad"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025511    4320 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025514    4320 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025517    4320 flags.go:64] FLAG: --register-node="true"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025520    4320 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025524    4320 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025527    4320 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025529    4320 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025533    4320 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025536    4320 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:04:12.033485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025539    4320 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025543    4320 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025546    4320 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025549    4320 flags.go:64] FLAG: --runonce="false"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025552    4320 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025555    4320 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025558    4320 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025561    4320 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025564    4320 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025567    4320 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025570    4320 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025573    4320 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025576    4320 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025579    4320 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025582    4320 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025585    4320 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025589    4320 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025592    4320 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025606    4320 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025612    4320 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025615    4320 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025618    4320 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025621    4320 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025624    4320 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025627    4320 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:04:12.034120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025630    4320 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025633    4320 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025636    4320 flags.go:64] FLAG: --v="2"
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025640    4320 flags.go:64] FLAG: --version="false"
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025644    4320 flags.go:64] FLAG: --vmodule=""
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025648    4320 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.025651    4320 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025777    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025781    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025784    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025787    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025790    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025793    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025795    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025798    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025801    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025803    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025806    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025808    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025811    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025814    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.034914 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025816    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025819    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025822    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025824    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025827    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025830    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025832    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025835    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025837    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025840    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025843    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025846    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025849    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025851    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025854    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025858    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025862    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025864    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025868    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.035793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025871    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025874    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025876    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025879    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025882    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025884    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025887    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025890    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025893    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025895    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025898    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025900    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025903    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025906    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025908    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025911    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025913    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025916    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025918    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025921    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.036438 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025924    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025926    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025929    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025931    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025934    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025936    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025939    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025941    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025944    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025946    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025949    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025951    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025954    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025956    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025959    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025962    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025965    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025967    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025970    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025973    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.037073 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025975    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025979    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025981    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025986    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025988    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025991    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025994    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025996    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.025999    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026002    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026004    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026008    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026013    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026016    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026020    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026024    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026027    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026029    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026032    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.037579 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026035    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026037    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026040    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026042    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026045    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026047    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026050    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026053    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026055    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026058    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026060    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026063    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026066    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026068    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.026092    4320 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026125    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.038187 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026130    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026134    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026137    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026140    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026142    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026145    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026148    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026150    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026154    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026157    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026160    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026162    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026165    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026168    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026170    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026173    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026176    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026178    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026181    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.038732 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026183    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026186    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026188    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026191    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026193    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026196    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026199    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026201    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026204    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026207    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026210    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026212    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026215    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026218    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026222    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026225    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026228    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026230    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026233    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026235    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.039245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026238    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026246    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026249    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026253    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026256    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026258    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026261    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026264    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026266    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026269    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026272    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026274    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026277    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026280    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026282    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026285    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026288    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026291    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026293    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026296    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.039790 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026298    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026301    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026303    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026306    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026308    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026311    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026315    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026318    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026321    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026323    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026326    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026328    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026331    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026336    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026338    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026341    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026343    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026346    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026349    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.040377 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026351    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026354    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026356    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026359    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026361    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026364    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026367    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026369    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026372    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026374    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026377    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026380    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026383    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026385    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026388    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026390    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026393    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026395    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026398    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.040867 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026404    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026409    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026412    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026416    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026419    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026421    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026424    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.026428    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.029850    4320 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.029863    4320 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.029880    4320 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.029884    4320 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029946    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029951    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029954    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029957    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029959    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029962    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029965    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.041363 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029968    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029971    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029974    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029976    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029979    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029982    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029984    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029987    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029990    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029992    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029995    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.029998    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030000    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030003    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030007    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030009    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030012    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030014    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030017    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.041850 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030021    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030025    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030028    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030032    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030035    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030038    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030040    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030043    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030046    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030049    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030051    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030054    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030057    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030059    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030062    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030065    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030067    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030070    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030072    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030075    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.042319 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030078    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030080    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030083    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030085    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030088    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030090    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030093    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030095    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030098    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030101    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030103    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030107    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030110    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030112    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030115    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030118    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030121    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030123    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030126    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.042842 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030128    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030131    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030133    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030136    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030138    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030141    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030144    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030146    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030149    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030151    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030154    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030157    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030159    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030162    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030166    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030169    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030172    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030175    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030178    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.043301 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030180    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030183    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030185    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030188    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030191    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030194    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030197    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030199    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030202    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030205    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030207    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030210    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030212    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030215    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030217    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030220    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030222    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030224    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030227    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030229    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.043780 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030232    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030235    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.030261    4320 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030390    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030395    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030398    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030401    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030403    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030406    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030408    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030411    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030413    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030416    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030419    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030422    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030424    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.044266 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030427    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030430    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030433    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030435    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030438    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030441    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030444    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030446    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030449    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030451    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030454    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030456    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030459    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030461    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030464    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030467    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030469    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030472    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030474    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030477    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.044667 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030479    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030482    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030484    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030487    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030489    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030492    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030494    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030497    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030499    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030502    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030504    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030507    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030510    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030513    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030516    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030519    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030522    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030524    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030527    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030530    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.045155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030533    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030535    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030538    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030540    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030542    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030545    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030548    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030550    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030553    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030555    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030558    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030560    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030563    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030565    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030568    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030570    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030573    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030575    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030578    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030581    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.045659 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030584    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030588    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030591    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030609    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030614    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030618    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030622    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030625    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030628    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030631    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030634    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030636    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030639    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030641    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030644    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030646    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030649    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030652    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030654    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030657    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.046135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030659    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030662    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030664    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030667    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030669    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030672    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030675    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030677    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030680    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030683    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030691    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030694    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030697    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.030721    4320 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030751    4320 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:04:12.046786 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030757    4320 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030760    4320 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030764    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030767    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030770    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030772    4320 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030775    4320 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030778    4320 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030781    4320 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030784    4320 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030786    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030789    4320 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030791    4320 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030794    4320 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030796    4320 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030799    4320 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030801    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030804    4320 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030807    4320 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030809    4320 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:04:12.047214 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030812    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030815    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030818    4320 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030822    4320 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030825    4320 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030827    4320 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030831    4320 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030833    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030837    4320 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030840    4320 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030842    4320 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030845    4320 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030849    4320 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030851    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030854    4320 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030856    4320 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030859    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030867    4320 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030870    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030872    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:04:12.047736 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030875    4320 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030877    4320 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030880    4320 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030882    4320 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030885    4320 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030887    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030890    4320 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030893    4320 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030895    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030898    4320 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030900    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030904    4320 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030908    4320 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030911    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030914    4320 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030917    4320 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030919    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030922    4320 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:04:12.048220 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030924    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030927    4320 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030931    4320 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030933    4320 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030936    4320 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030939    4320 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030941    4320 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030944    4320 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030947    4320 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030949    4320 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030952    4320 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030956    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030960    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030962    4320 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030965    4320 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030967    4320 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030970    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030972    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030975    4320 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030977    4320 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:04:12.048864 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030980    4320 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030982    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030985    4320 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030987    4320 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030990    4320 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030992    4320 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030994    4320 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030997    4320 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.030999    4320 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031002    4320 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031005    4320 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031007    4320 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031010    4320 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031012    4320 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031016    4320 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031019    4320 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031022    4320 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031025    4320 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031028    4320 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:04:12.049353 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031030    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031033    4320 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031038    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031040    4320 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031045    4320 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031048    4320 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031051    4320 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:12.031053    4320 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.031099    4320 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032101    4320 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032156    4320 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032320    4320 server.go:1014] "Starting client certificate rotation"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032473    4320 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032517    4320 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:33 +0000 UTC" deadline="2028-05-03 18:10:04.653227225 +0000 UTC"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032562    4320 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13223h5m52.620668336s"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.032828    4320 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:04:12.049870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.034239    4320 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:04:12.050314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.036133    4320 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:04:12.050314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.037667    4320 log.go:25] "Validated CRI v1 image API"
Oct 30 19:04:12.050314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.038429    4320 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:04:12.050314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.039696    4320 fs.go:135] Filesystem UUIDs: map[12109c07-cbd9-4dde-af9c-ce22653c9b99:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d6654752-17f9-43db-9dcd-86ac1f1396bb:/dev/nvme0n1p3]
Oct 30 19:04:12.050430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.039717    4320 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0261b7c5-c9bb-4fb6-aa6d-23914249b848/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0261b7c5-c9bb-4fb6-aa6d-23914249b848/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~projected/kube-api-access-8f2pr:{mountpoint:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~projected/kube-api-access-8f2pr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2357170c-bfb3-4ee9-925f-e84d351a2c86/volumes/kubernetes.io~projected/kube-api-access-8kx7m:{mountpoint:/var/lib/kubelet/pods/2357170c-bfb3-4ee9-925f-e84d351a2c86/volumes/kubernetes.io~projected/kube-api-access-8kx7m major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~projected/kube-api-access-sdp6x:{mountpoint:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~projected/kube-api-access-sdp6x major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c416fac-c47f-487a-94d8-d27d779ad0d5/volumes/kubernetes.io~projected/kube-api-access-xzdvb:{mountpoint:/var/lib/kubelet/pods/4c416fac-c47f-487a-94d8-d27d779ad0d5/volumes/kubernetes.io~projected/kube-api-access-xzdvb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aba8d2a-7b5b-4265-b8ca-48cac068159d/volumes/kubernetes.io~projected/kube-api-access-zbvx8:{mountpoint:/var/lib/kubelet/pods/7aba8d2a-7b5b-4265-b8ca-48cac068159d/volumes/kubernetes.io~projected/kube-api-access-zbvx8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b56513e8-bfbe-4662-a549-6aaf87a53316/volumes/kubernetes.io~projected/kube-api-access-fk7v7:{mountpoint:/var/lib/kubelet/pods/b56513e8-bfbe-4662-a549-6aaf87a53316/volumes/kubernetes.io~projected/kube-api-access-fk7v7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0885c92-5132-4361-901c-4530f23a5c57/volumes/kubernetes.io~projected/kube-api-access-5jnvt:{mountpoint:/var/lib/kubelet/pods/c0885c92-5132-4361-901c-4530f23a5c57/volumes/kubernetes.io~projected/kube-api-access-5jnvt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4b52d98-fb0b-4145-aaf5-dc3301172e4b/volumes/kubernetes.io~projected/kube-api-access-ljhbd:{mountpoint:/var/lib/kubelet/pods/c4b52d98-fb0b-4145-aaf5-dc3301172e4b/volumes/kubernetes.io~projected/kube-api-access-ljhbd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7/volumes/kubernetes.io~projected/kube-api-access-c7w6f:{mountpoint:/var/lib/kubelet/pods/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7/volumes/kubernetes.io~projected/kube-api-access-c7w6f major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbc31816-a783-44bd-83e4-a009e13da8b4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cbc31816-a783-44bd-83e4-a009e13da8b4/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d54891c7-b195-47ad-8bee-911c66922826/volumes/kubernetes.io~projected/kube-api-access-l59h2:{mountpoint:/var/lib/kubelet/pods/d54891c7-b195-47ad-8bee-911c66922826/volumes/kubernetes.io~projected/kube-api-access-l59h2 major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/18326f631b2b065fc60518a590aebcd3dc77e09d25cd4212b371f6dd28b53813/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/96a2531788915fef562c6be3ab05e4ff5170072fb065ea09c2cc456e87cd73ea/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ee24510eb9de82c6940aa117a896566b744fe0c2ebc6edade03f471468e9cf14/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/09871de1a884b4fdaaf5575d98c5be67c557679d1103f6a8ee5b2905f5e00869/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7f67db0b42b9264564d188cd0ee0ac6f63d26b45ff393ff0bea44eb020687053/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fe46e88528779b72c5ffdb1810f30bd7d8a5293f166c9f3b222f48301506f719/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0484120e14b30837f9169e299b0a69d3005c274cdbc057ab7496d1481e3d968f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a44b211459062a86885662bcd14cf74ab4e95d701c8efdbfaafb6f213b423b30/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/2477a3e977917c5785609b3e26879e3a104d8171578b8eb808e28c4e02f9bd27/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/765b7ed6e0e07a66ea6552e9ac1e0a045e0b2d84b36c68df07cb1fdb842401ca/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/56f3cc25f333db3257cc00f68455a60c3c021c2ffc944a7d47823faa326d60ec/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b73146604aa3ec5a889cac7f4db3cd199dc6975da7d3e071d17b16d18e13ecf2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/972e22763afeef8b8c0c19874ce97fed1b96cb1d2c3cd8242f3667a12c96c99d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8fb686779bf83f5af526f4b89440837216c3a2ab97a0a2a172172d6b61e044c8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bbd790cc5f6f1a3048762aedc88561e692c169571fceb40ad842f6eedc29675b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8ae098825046e0fcf42a38d9a7de8adf56408b1223965b0e6a986f5edb5778b0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/72e7336eb52e5d809472463baff9d0695274d2338237f6beb90c2e9ae0e996c1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0312cf41cb3674b94b45c2ba1ca9efe3b2c8822ac1618057654a4ad12827814d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c112aa7c6c4ba6db6943476004528c7d46ab08c5200a8f064a74f3b7e62e99b0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/35e54334da3803c40c54c387a7da88dd354ac153cfdd79d29cf336894cadf82f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e54085b62ab05f5e04bec6ee59edd281ec69960eb77582c39518020acc244b4f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/a9a2b5b95f66ce61e8ea162fe27e060198bf378ce959f15b55c7dc9ad8a3225b/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/f39372912f6799f13f6307a801240119aba578c1d5ac458ef5f9ec70a071834c/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b29d3d690cf3be4350c8a194b1087c7c05e8c086aa495db206f50825523d1dc0/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f89ccda73aa20e95b14ee11aaea4bef10e430e0e1e3e1478c68789bbe11abf4c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/d9b1e4780fcd0ab0cfd5fd3ea5990c3ceda1299cc1c098e9dc10dda83fd07205/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/227cd3ddb1d9b19823798d94b4e1e60f334536733478c35ba153eb2331d83ff8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0a6333182667e11a7d142afd2f4aebbcfdda10543f0d04352fb5569ebfd7f34a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/61148cd8068f0522e046352e4182093403292423f460e2720967ddb7f7f25de0/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ca68df8eaff9da9012d8c3cd4f0c1d170d2d7cc1185f18c830db2b4d1e88ab22/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/bafab8392ab6e744080b50390d1d9f0a91863c7725c11e6209220307a63b4c53/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/9ba3bf63214c60438c3e344cdb5f2542334caf885f62275371baa56468e9168e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/aebce80afff0a959d3b1fc594279c1389f8f6044b86609ed1d47e7c020c4ae2d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/1070a6e780cbeac4e4302c7ecf184f3c324c0c3c761f669cf2dc0470f3211f80/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/da0a82998528e1f4d2ce73964c599982e40af0339929085054bdd0f67ebc3ecf/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045154    4320 manager.go:217] Machine: {Timestamp:2026-10-30 19:04:12.044711179 +0000 UTC m=+0.080223634 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099910 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268791b75f0d4fdfe6e0a5a85096fa SystemUUID:ec268791-b75f-0d4f-dfe6-e0a5a85096fa BootID:0a73bff7-c4e9-48dd-b8fe-5440ed31a46f Filesystems:[{Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e21ce74d6e2d5db7b9cc6b1cd64c4712b62acebb015aa34f285defcb99fa1187/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0261b7c5-c9bb-4fb6-aa6d-23914249b848/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d54891c7-b195-47ad-8bee-911c66922826/volumes/kubernetes.io~projected/kube-api-access-l59h2 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbc31816-a783-44bd-83e4-a009e13da8b4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca6d36c3c4ec4def6c98e066559df407b296b9f097e1a7eae0535ad64442f8fa/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~projected/kube-api-access-8f2pr DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1a72ab05b81dd67f771c5a07e270d448ab2d2caa3092302c6fd8076ab36f663/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~projected/kube-api-access-sdp6x DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c4b52d98-fb0b-4145-aaf5-dc3301172e4b/volumes/kubernetes.io~projected/kube-api-access-ljhbd DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2357170c-bfb3-4ee9-925f-e84d351a2c86/volumes/kubernetes.io~projected/kube-api-access-8kx7m DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a4cbfa7dbcec3fc3a7feb46734456ef1096f3727171408d642a41ff99cdb551/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffe9559700680967d96a0a35496a1c590b7f10bd07f9f5b0b05b67a2032d41ff/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9ace0519176db546517d08a71d77c3690aba77ae61bb3d9129f886f7e79de4c/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a178aeea6c33e038d0234b28b9c58185406a4d7186154c30a06fda736e9f09d/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/638aac018121bcc39caa70231c29f064ec0682159ba3edcd9e46c1c9d9f28854/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0885c92-5132-4361-901c-4530f23a5c57/volumes/kubernetes.io~projected/kube-api-access-5jnvt DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b56513e8-bfbe-4662-a549-6aaf87a53316/volumes/kubernetes.io~projected/kube-api-access-fk7v7 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/393d567d618f819f7b488e890404f959949517ae9b6a266a12ef2e6e60dee316/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7/volumes/kubernetes.io~projected/kube-api-access-c7w6f DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49c65674-cd6d-45e7-a7ad-2469ca4910cb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c416fac-c47f-487a-94d8-d27d779ad0d5/volumes/kubernetes.io~projected/kube-api-access-xzdvb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b28067397d799d153aa0b00d2c54e2c70dbd9bea1b50a814e579ab9f463ed856/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aba8d2a-7b5b-4265-b8ca-48cac068159d/volumes/kubernetes.io~projected/kube-api-access-zbvx8 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:b28067397d799d1 MacAddress:ce:83:b1:b0:1e:58 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:c2:c3:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:da:33:7c:75:a3 Speed:0 Mtu:8901} {Name:ca6d36c3c4ec4de MacAddress:46:7c:b4:f7:0e:b3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:c2:c3:c1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:6c:b8:c5:e4:14 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:c5:bf:93:17:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045366    4320 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045403    4320 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045660    4320 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045683    4320 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045806    4320 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045814    4320 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045823    4320 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045835    4320 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.045891    4320 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046024    4320 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046121    4320 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046151    4320 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046164    4320 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046173    4320 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046181    4320 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046189    4320 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046691    4320 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.046700    4320 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.047150    4320 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049121    4320 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049130    4320 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049211    4320 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049374    4320 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049400    4320 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049478    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049490    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049495    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049501    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049507    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049514    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049520    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049525    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049531    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049537    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049551    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049560    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049583    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:04:12.050559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.049589    4320 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:04:12.052645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.052633    4320 server.go:1263] "Started kubelet"
Oct 30 19:04:12.053317 ip-10-0-130-217 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:04:12.056152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.053491    4320 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:04:12.056152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.053532    4320 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:04:12.056152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.054109    4320 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:04:12.056152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.054366    4320 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:04:12.056452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.056435    4320 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:04:12.057736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.057695    4320 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:04:12.057736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.057712    4320 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:04:12.058194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.057943    4320 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:35 +0000 UTC" deadline="2028-05-05 21:14:21.593892683 +0000 UTC"
Oct 30 19:04:12.058194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.057981    4320 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13274h10m9.535915293s"
Oct 30 19:04:12.058194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.058110    4320 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:04:12.058194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.058118    4320 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:04:12.058194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.058185    4320 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:04:12.058899 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.058457    4320 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-217.ec2.internal\" not found"
Oct 30 19:04:12.059207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059004    4320 factory.go:55] Registering systemd factory
Oct 30 19:04:12.059207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059023    4320 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:04:12.059346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059276    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0261b7c5-c9bb-4fb6-aa6d-23914249b848" volumeName="kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret" seLinuxMountContext=""
Oct 30 19:04:12.059346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059316    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb84c0-12d1-4c7b-9001-42a3f6b1391a" volumeName="kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:04:12.059346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059332    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2357170c-bfb3-4ee9-925f-e84d351a2c86" volumeName="kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059346    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aba8d2a-7b5b-4265-b8ca-48cac068159d" volumeName="kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059359    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b56513e8-bfbe-4662-a549-6aaf87a53316" volumeName="kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059381    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0885c92-5132-4361-901c-4530f23a5c57" volumeName="kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059393    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0885c92-5132-4361-901c-4530f23a5c57" volumeName="kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059417    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b52d98-fb0b-4145-aaf5-dc3301172e4b" volumeName="kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059454    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb84c0-12d1-4c7b-9001-42a3f6b1391a" volumeName="kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059468    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb84c0-12d1-4c7b-9001-42a3f6b1391a" volumeName="kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config" seLinuxMountContext=""
Oct 30 19:04:12.059485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059481    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb84c0-12d1-4c7b-9001-42a3f6b1391a" volumeName="kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059527    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c416fac-c47f-487a-94d8-d27d779ad0d5" volumeName="kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059542    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0885c92-5132-4361-901c-4530f23a5c57" volumeName="kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059556    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0885c92-5132-4361-901c-4530f23a5c57" volumeName="kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059573    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc31816-a783-44bd-83e4-a009e13da8b4" volumeName="kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059585    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc31816-a783-44bd-83e4-a009e13da8b4" volumeName="kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059693    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb84c0-12d1-4c7b-9001-42a3f6b1391a" volumeName="kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059702    4320 factory.go:153] Registering CRI-O factory
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059706    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2357170c-bfb3-4ee9-925f-e84d351a2c86" volumeName="kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059717    4320 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059720    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b56513e8-bfbe-4662-a549-6aaf87a53316" volumeName="kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059780    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca3c9c4d-c2a2-4fb7-a250-396d97523ad7" volumeName="kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059793    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d54891c7-b195-47ad-8bee-911c66922826" volumeName="kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059805    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c65674-cd6d-45e7-a7ad-2469ca4910cb" volumeName="kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059819    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c65674-cd6d-45e7-a7ad-2469ca4910cb" volumeName="kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059831    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49c65674-cd6d-45e7-a7ad-2469ca4910cb" volumeName="kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059842    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c416fac-c47f-487a-94d8-d27d779ad0d5" volumeName="kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059862    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b52d98-fb0b-4145-aaf5-dc3301172e4b" volumeName="kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059767    4320 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059875    4320 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b52d98-fb0b-4145-aaf5-dc3301172e4b" volumeName="kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059887    4320 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:04:12.059884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059895    4320 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:04:12.060820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.059894    4320 factory.go:103] Registering Raw factory
Oct 30 19:04:12.060820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.060007    4320 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:04:12.060820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.060384    4320 manager.go:319] Starting recovery of all containers
Oct 30 19:04:12.086976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.086948    4320 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:04:12.088174 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.088152    4320 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:04:12.088249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.088177    4320 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:04:12.088249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.088199    4320 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:04:12.088249 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.088238    4320 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:04:12.104522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.104499    4320 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:04:12.105782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.105751    4320 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:04:12.105866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.105842    4320 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:04:12.106325 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.106305    4320 generic.go:356] "Generic (PLEG): container finished" podID="21287f518f1b139efdbea74cc5c91860" containerID="b18f263eceb2052f3f3d194ec4411496f763a2f7e4fb82e73198c1338c9bc877" exitCode=0
Oct 30 19:04:12.116791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.116771    4320 manager.go:324] Recovery completed
Oct 30 19:04:12.118139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.118119    4320 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:04:12.118239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.118216    4320 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:04:12.119014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.118992    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="090e60d5e877fdc43a9b225287efd2d72e213ff764e3a44aa9095861856e4db5" exitCode=0
Oct 30 19:04:12.119014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.119014    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="704c3e3533233abde9e6f46a3bb33cda4d80584922049f02e2fd8e9a135809c5" exitCode=0
Oct 30 19:04:12.119150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.119023    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="9c3f2b663fef8bd42711bbf26eaa53a93550c2692e6724a8c9fd36493c0803c0" exitCode=0
Oct 30 19:04:12.119150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.119028    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="a68655a4d89a39c2db1e29975ec6d2a5576f1bfd4aa8c604e3176da649ae3bcc" exitCode=0
Oct 30 19:04:12.119150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.119033    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="5437eb0edad2ee97a44c7408020005159613f9dfbbd162246e1aa135d598d582" exitCode=0
Oct 30 19:04:12.119150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.119038    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0885c92-5132-4361-901c-4530f23a5c57" containerID="64cb33f98c0108cd8f577a3eef5d161d983c37814a8231b30747ed3161a31487" exitCode=0
Oct 30 19:04:12.129441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129423    4320 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:04:12.129441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129437    4320 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:04:12.129548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129456    4320 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:04:12.129631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129591    4320 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:04:12.129676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129633    4320 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:04:12.129716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129677    4320 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:04:12.129716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129683    4320 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:04:12.129716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129688    4320 policy_none.go:50] "Start"
Oct 30 19:04:12.129716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129697    4320 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:04:12.129716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129708    4320 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:04:12.129868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129825    4320 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:04:12.129868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129831    4320 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:04:12.129868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.129836    4320 policy_none.go:44] "Start"
Oct 30 19:04:12.133588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.133562    4320 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:04:12.136087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.136072    4320 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:04:12.136565 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.136094    4320 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:04:12.136565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.136565    4320 server.go:87] "Starting device plugin registration server"
Oct 30 19:04:12.136815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.136802    4320 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:04:12.136877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.136815    4320 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:04:12.136954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.136920    4320 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:04:12.137090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.137020    4320 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:04:12.137090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.137056    4320 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:04:12.137090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.137063    4320 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:04:12.137237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.137110    4320 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:04:12.137046783Z","UUID":"3c389ada-0a9f-4dc1-9eb3-358599fe9c6c","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:04:12.138074 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.138008    4320 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:04:12.138642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.138581    4320 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:04:12.138642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.138619    4320 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:04:12.188552 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.188487    4320 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"]
Oct 30 19:04:12.188965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.188938    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" event={"ID":"32feda196b1df490b5d951791a17beab","Type":"ContainerStarted","Data":"82638b06bc82a1dc45bae9c6af28814d9985cc902c21cd55d0c5694522e419fa"}
Oct 30 19:04:12.189009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.188972    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" event={"ID":"32feda196b1df490b5d951791a17beab","Type":"ContainerStarted","Data":"a9d98f4c71fdbf330f49de043375eaa1de6b720fd9d4150577ef968b68e345f4"}
Oct 30 19:04:12.189059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.189025    4320 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.189091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.189076    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerStarted","Data":"e42e3855ec68ed131fe72f585dd7606437cf97e6178cca90c8d4d67e5112dccf"}
Oct 30 19:04:12.189135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.189098    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerDied","Data":"b18f263eceb2052f3f3d194ec4411496f763a2f7e4fb82e73198c1338c9bc877"}
Oct 30 19:04:12.189135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.189115    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal" event={"ID":"21287f518f1b139efdbea74cc5c91860","Type":"ContainerStarted","Data":"884eff8418597521d5483c89f6ab9b09dce1422780ebbf70eaec0075d8167e99"}
Oct 30 19:04:12.189135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.189123    4320 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.210791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.210773    4320 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:04:12.210856 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.210811    4320 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.211288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.211276    4320 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:04:12.211323 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:12.211308    4320 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.237422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.237402    4320 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:04:12.238557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.238542    4320 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:04:12.238629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.238567    4320 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:04:12.238629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.238580    4320 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-217.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:04:12.238629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.238628    4320 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.275975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.275956    4320 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.276070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.276012    4320 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.306694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.306674    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.306786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.306699    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.306786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.306716    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.406981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.406960    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.407078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.406987    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.407078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.407004    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.407078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.407043    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32feda196b1df490b5d951791a17beab-config\") pod \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" (UID: \"32feda196b1df490b5d951791a17beab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.407193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.407078    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:12.407193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:12.407080    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21287f518f1b139efdbea74cc5c91860-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" (UID: \"21287f518f1b139efdbea74cc5c91860\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:13.046649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.046622    4320 apiserver.go:52] "Watching apiserver"
Oct 30 19:04:13.086194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.086169    4320 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:04:13.086538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.086518    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kbslw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb","openshift-image-registry/node-ca-6wmz8","openshift-multus/multus-additional-cni-plugins-gq2v8","kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal","openshift-dns/node-resolver-ckfxz","openshift-network-diagnostics/network-check-target-r5zck","openshift-network-operator/iptables-alerter-lxwvs","kube-system/konnectivity-agent-rrwhk","openshift-cluster-node-tuning-operator/tuned-hxrgs","openshift-multus/multus-nppsv","openshift-multus/network-metrics-daemon-kn5zp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-kgjxh"]
Oct 30 19:04:13.092677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.092650    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.102663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102614    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.102663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102638    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.102663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102642    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102686    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102728    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102774    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102782    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102791    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102823    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102825    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102835    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:04:13.102869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.102856    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\""
Oct 30 19:04:13.124075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.124060    4320 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:13.124188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.124176    4320 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:13.137604 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.137584    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.137891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.137874    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:04:13.138180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138164    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.138352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138340    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:04:13.138411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138373    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.138467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138455    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.138715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138700    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.138805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138720    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.138805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138729    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.138805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138742    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:04:13.138805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138759    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:04:13.138944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138816    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:04:13.138944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138931    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.139027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138949    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:04:13.139027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138970    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:04:13.139027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138981    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:04:13.139027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.138988    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:04:13.146194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.146179    4320 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:04:13.146237 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:13.146223    4320 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-217.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:13.146478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.146466    4320 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:04:13.146518 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:13.146497    4320 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-217.ec2.internal"
Oct 30 19:04:13.159467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.159448    4320 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:04:13.203815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.203795    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\""
Oct 30 19:04:13.204011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.203997    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:04:13.204084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204070    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:04:13.204128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204088    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\""
Oct 30 19:04:13.204128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204088    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\""
Oct 30 19:04:13.204291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204277    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\""
Oct 30 19:04:13.204556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204529    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204544    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204563    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204580    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204626    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204636    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\""
Oct 30 19:04:13.204658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.204642    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\""
Oct 30 19:04:13.205031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.205020    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\""
Oct 30 19:04:13.212054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212034    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.212150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212058    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.212150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212084    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.212150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212110    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.212150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212133    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.212396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212166    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.212396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212382    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.212455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212425    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.212484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212455    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.212484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212477    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kx7m\" (UniqueName: \"kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.212553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212505    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.212553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212513    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.212666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212648    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:13.212715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212669    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.212715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212693    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:04:13.212790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212739    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.212821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212804    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.212860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212834    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.212906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212876    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.212944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212918    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.212973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212943    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.212973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212964    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.213062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.212998    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b56513e8-bfbe-4662-a549-6aaf87a53316-serviceca\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.213062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213008    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp6x\" (UniqueName: \"kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.213062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213051    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-daemon-config\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213085    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cbc31816-a783-44bd-83e4-a009e13da8b4-agent-certs\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:04:13.213201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213162    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213185    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213203    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.213340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213237    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.213340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213281    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.213340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213320    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213343    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213359    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213377    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdvb\" (UniqueName: \"kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213398    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213426    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213435    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0261b7c5-c9bb-4fb6-aa6d-23914249b848-original-pull-secret\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213447    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvx8\" (UniqueName: \"kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213461    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.213530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213494    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213537    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213571    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213638    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213658    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213662    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-env-overrides\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213674    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213694    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213798    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213828    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213850    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213874    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.213911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213902    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213931    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213944    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c416fac-c47f-487a-94d8-d27d779ad0d5-iptables-alerter-script\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213957    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.213981    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214004    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214043    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214103    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214114    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-tuned\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214142    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-config\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214146    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovn-node-metrics-cert\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214170    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2357170c-bfb3-4ee9-925f-e84d351a2c86-tmp-dir\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214173    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214165    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cni-binary-copy\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214207    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214259    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214304    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214344    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.214414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214388    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214426    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-ovnkube-script-lib\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214442    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49c65674-cd6d-45e7-a7ad-2469ca4910cb-tmp\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214445    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214545    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cbc31816-a783-44bd-83e4-a009e13da8b4-konnectivity-ca\") pod \"konnectivity-agent-rrwhk\" (UID: \"cbc31816-a783-44bd-83e4-a009e13da8b4\") " pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214570    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214622    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhbd\" (UniqueName: \"kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214652    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214685    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnvt\" (UniqueName: \"kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214712    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214735    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214760    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214794    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2pr\" (UniqueName: \"kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214829    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214872    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0885c92-5132-4361-901c-4530f23a5c57-cni-binary-copy\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214924    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214951    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.214991    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w6f\" (UniqueName: \"kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215023    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215045    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215068    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215108    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215132    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215175    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215222    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215246    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7v7\" (UniqueName: \"kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.215658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.215259    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.230051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.230012    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp6x\" (UniqueName: \"kubernetes.io/projected/49c65674-cd6d-45e7-a7ad-2469ca4910cb-kube-api-access-sdp6x\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.230147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.230134    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kx7m\" (UniqueName: \"kubernetes.io/projected/2357170c-bfb3-4ee9-925f-e84d351a2c86-kube-api-access-8kx7m\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.230271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.230241    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59h2\" (UniqueName: \"kubernetes.io/projected/d54891c7-b195-47ad-8bee-911c66922826-kube-api-access-l59h2\") pod \"network-check-target-r5zck\" (UID: \"d54891c7-b195-47ad-8bee-911c66922826\") " pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:13.230467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.230452    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdvb\" (UniqueName: \"kubernetes.io/projected/4c416fac-c47f-487a-94d8-d27d779ad0d5-kube-api-access-xzdvb\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.230513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.230502    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvx8\" (UniqueName: \"kubernetes.io/projected/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kube-api-access-zbvx8\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.231280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.231264    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhbd\" (UniqueName: \"kubernetes.io/projected/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-kube-api-access-ljhbd\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.231683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.231656    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnvt\" (UniqueName: \"kubernetes.io/projected/c0885c92-5132-4361-901c-4530f23a5c57-kube-api-access-5jnvt\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.231758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.231683    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7v7\" (UniqueName: \"kubernetes.io/projected/b56513e8-bfbe-4662-a549-6aaf87a53316-kube-api-access-fk7v7\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.231758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.231721    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w6f\" (UniqueName: \"kubernetes.io/projected/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-kube-api-access-c7w6f\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.231833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.231795    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2pr\" (UniqueName: \"kubernetes.io/projected/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-kube-api-access-8f2pr\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316156    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316183    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316198    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316216    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316239    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316264    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316263    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-systemd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316275    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-netns\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316280    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-var-lib-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316318    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316319    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-bin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316334    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.316379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316382    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316386    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-conf\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316394    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-multus-certs\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316457    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316457    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-kubernetes\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316492    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316523    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-cnibin\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316553    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316559    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-conf-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316571    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316587    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316628    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-etc-kubernetes\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316633    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316647    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316663    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316679    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316715    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-modprobe-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316724    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316728    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-hostroot\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316725    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316744    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-k8s-cni-cncf-io\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316753    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.316771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316771    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316793    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316795    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316807    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-netd\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316813    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-host\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316829    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316844    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316853    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316877    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316900    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316916    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-kubelet-config\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316906    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-cnibin\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316929    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316943    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-device-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316953    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316944    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316978    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-node-log\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.316986    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317003    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317019    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317033    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysctl-d\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317054    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b56513e8-bfbe-4662-a549-6aaf87a53316-host\") pod \"node-ca-6wmz8\" (UID: \"b56513e8-bfbe-4662-a549-6aaf87a53316\") " pod="openshift-image-registry/node-ca-6wmz8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317033    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317069    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-run\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317076    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-systemd-units\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317078    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317093    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-run-ovn\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317100    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317115    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317129    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317142    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317167    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317173    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0885c92-5132-4361-901c-4530f23a5c57-system-cni-dir\") pod \"multus-additional-cni-plugins-gq2v8\" (UID: \"c0885c92-5132-4361-901c-4530f23a5c57\") " pod="openshift-multus/multus-additional-cni-plugins-gq2v8"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317180    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-systemd\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317187    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0261b7c5-c9bb-4fb6-aa6d-23914249b848-dbus\") pod \"global-pull-secret-syncer-kbslw\" (UID: \"0261b7c5-c9bb-4fb6-aa6d-23914249b848\") " pod="kube-system/global-pull-secret-syncer-kbslw"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317189    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317201    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317207    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c416fac-c47f-487a-94d8-d27d779ad0d5-host-slash\") pod \"iptables-alerter-lxwvs\" (UID: \"4c416fac-c47f-487a-94d8-d27d779ad0d5\") " pod="openshift-network-operator/iptables-alerter-lxwvs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317220    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317220    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317241    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317249    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-sys\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317220    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-kubelet\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317317    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-slash\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317319    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317343    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317347    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-etc-openvswitch\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317382    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317403    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-cni-bin\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317406    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317385    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aba8d2a-7b5b-4265-b8ca-48cac068159d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tzfb\" (UID: \"7aba8d2a-7b5b-4265-b8ca-48cac068159d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tzfb"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317421    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317431    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-run-netns\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317437    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317452    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317456    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-etc-sysconfig\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317468    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317483    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-var-lib-kubelet\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317484    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317506    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.317490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317511    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-kubelet\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317518    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317568    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317615    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317632    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-multus-socket-dir-parent\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317644    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317669    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aeb84c0-12d1-4c7b-9001-42a3f6b1391a-log-socket\") pod \"ovnkube-node-kgjxh\" (UID: \"0aeb84c0-12d1-4c7b-9001-42a3f6b1391a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317676    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49c65674-cd6d-45e7-a7ad-2469ca4910cb-lib-modules\") pod \"tuned-hxrgs\" (UID: \"49c65674-cd6d-45e7-a7ad-2469ca4910cb\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrgs"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317667    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317707    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-system-cni-dir\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317708    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4b52d98-fb0b-4145-aaf5-dc3301172e4b-host-var-lib-cni-multus\") pod \"multus-nppsv\" (UID: \"c4b52d98-fb0b-4145-aaf5-dc3301172e4b\") " pod="openshift-multus/multus-nppsv"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317715    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317773    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2357170c-bfb3-4ee9-925f-e84d351a2c86-hosts-file\") pod \"node-resolver-ckfxz\" (UID: \"2357170c-bfb3-4ee9-925f-e84d351a2c86\") " pod="openshift-dns/node-resolver-ckfxz"
Oct 30 19:04:13.319352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.317828    4320 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:04:13.319820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.319806    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca3c9c4d-c2a2-4fb7-a250-396d97523ad7-metrics-certs\") pod \"network-metrics-daemon-kn5zp\" (UID: \"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7\") " pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.393520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.393504    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kn5zp"
Oct 30 19:04:13.722993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.722956    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kn5zp"]
Oct 30 19:04:13.727734 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:13.727705    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3c9c4d_c2a2_4fb7_a250_396d97523ad7.slice/crio-40a5eae26026f8c03a5e81b73231a5ac2cdc11b487ecade5f725a04495ba6021 WatchSource:0}: Error finding container 40a5eae26026f8c03a5e81b73231a5ac2cdc11b487ecade5f725a04495ba6021: Status 404 returned error can't find the container with id 40a5eae26026f8c03a5e81b73231a5ac2cdc11b487ecade5f725a04495ba6021
Oct 30 19:04:13.729981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.729961    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:04:13.777790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:13.777767    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:14.127463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:14.127398    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kn5zp" event={"ID":"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7","Type":"ContainerStarted","Data":"40a5eae26026f8c03a5e81b73231a5ac2cdc11b487ecade5f725a04495ba6021"}
Oct 30 19:04:15.131134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:15.131097    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kn5zp" event={"ID":"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7","Type":"ContainerStarted","Data":"b1299fdd56ed56a136a0a0dc3230b35be21801720b663a8dc51f2f2f8d33d379"}
Oct 30 19:04:15.131461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:15.131166    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kn5zp" event={"ID":"ca3c9c4d-c2a2-4fb7-a250-396d97523ad7","Type":"ContainerStarted","Data":"185c5411a9e36ff235a20705954dce0e5dd7721a5be3caa0993c5a0d30b2842f"}
Oct 30 19:04:20.661717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:20.661685    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:22.142947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:22.142918    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:22.143398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:22.143112    4320 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:04:23.887502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.887473    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"]
Oct 30 19:04:23.889303 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.889286    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:23.911627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.911588    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zlvtb\""
Oct 30 19:04:23.911736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.911718    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 19:04:23.922005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.921983    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"]
Oct 30 19:04:23.953245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.953211    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2nn6"]
Oct 30 19:04:23.954813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.954794    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:23.959870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.959848    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-529p7"]
Oct 30 19:04:23.961766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.961743    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:23.966261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.966235    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be020691-8510-4e02-9ee5-d01d497d6710-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-56cvd\" (UID: \"be020691-8510-4e02-9ee5-d01d497d6710\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:23.973581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.973563    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:04:23.977467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.977447    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-89vxf"]
Oct 30 19:04:23.979941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.979924    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:23.980841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.980801    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:04:23.983457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.983435    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2nn6"]
Oct 30 19:04:23.988937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.988918    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-529p7"]
Oct 30 19:04:23.989188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.989170    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cvd84"]
Oct 30 19:04:23.990612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.990580    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:04:23.991116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.991098    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:23.999996 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:23.999978    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:04:24.000074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.000040    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:04:24.010583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.010556    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89vxf"]
Oct 30 19:04:24.013675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.013657    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbgr5\""
Oct 30 19:04:24.013778 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.013753    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:04:24.013834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.013811    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:04:24.013998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.013984    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9w5x\""
Oct 30 19:04:24.014147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.014126    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-mjp8k\""
Oct 30 19:04:24.014217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.014123    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bwrqq\""
Oct 30 19:04:24.014274 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.014214    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:04:24.015318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.015299    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvd84"]
Oct 30 19:04:24.022455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.022440    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:04:24.022547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.022455    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:04:24.022588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.022564    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:04:24.050793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.050774    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77469d6cc6-zjsth"]
Oct 30 19:04:24.052432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.052416    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.067367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067350    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklq7\" (UniqueName: \"kubernetes.io/projected/b81a300d-dfb4-4853-b354-e541e6c0aec4-kube-api-access-sklq7\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.067461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067373    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6edeecf4-b1fb-42bf-8a2a-f0c379443841-tmp\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.067461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067391    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87615c5-a046-4c52-8d19-df38cbb27340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.067461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067409    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87615c5-a046-4c52-8d19-df38cbb27340-data-volume\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.067461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067424    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81a300d-dfb4-4853-b354-e541e6c0aec4-metrics-tls\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.067693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067483    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87615c5-a046-4c52-8d19-df38cbb27340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.067693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067532    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5kh\" (UniqueName: \"kubernetes.io/projected/a87615c5-a046-4c52-8d19-df38cbb27340-kube-api-access-5r5kh\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.067693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067632    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be020691-8510-4e02-9ee5-d01d497d6710-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-56cvd\" (UID: \"be020691-8510-4e02-9ee5-d01d497d6710\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:24.067693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067666    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52rc\" (UniqueName: \"kubernetes.io/projected/6edeecf4-b1fb-42bf-8a2a-f0c379443841-kube-api-access-z52rc\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.067855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067697    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81a300d-dfb4-4853-b354-e541e6c0aec4-tmp-dir\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.067855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067745    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntb7\" (UniqueName: \"kubernetes.io/projected/59938fcc-4816-4ad1-9a47-413063935cb6-kube-api-access-pntb7\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.067855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067807    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b81a300d-dfb4-4853-b354-e541e6c0aec4-config-volume\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.067855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067827    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59938fcc-4816-4ad1-9a47-413063935cb6-cert\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.067855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.067846    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87615c5-a046-4c52-8d19-df38cbb27340-crio-socket\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.070023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.070007    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/be020691-8510-4e02-9ee5-d01d497d6710-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-56cvd\" (UID: \"be020691-8510-4e02-9ee5-d01d497d6710\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:24.081179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.081152    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:04:24.081751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.081726    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77469d6cc6-zjsth"]
Oct 30 19:04:24.117911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.117887    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:04:24.118515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.118485    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ghkqg\""
Oct 30 19:04:24.118626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.118518    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:04:24.118811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.118786    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:04:24.168679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168628    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81a300d-dfb4-4853-b354-e541e6c0aec4-metrics-tls\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.168679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168656    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87615c5-a046-4c52-8d19-df38cbb27340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.168679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168674    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5kh\" (UniqueName: \"kubernetes.io/projected/a87615c5-a046-4c52-8d19-df38cbb27340-kube-api-access-5r5kh\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168698    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52rc\" (UniqueName: \"kubernetes.io/projected/6edeecf4-b1fb-42bf-8a2a-f0c379443841-kube-api-access-z52rc\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168727    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81a300d-dfb4-4853-b354-e541e6c0aec4-tmp-dir\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168756    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntb7\" (UniqueName: \"kubernetes.io/projected/59938fcc-4816-4ad1-9a47-413063935cb6-kube-api-access-pntb7\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168785    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-bound-sa-token\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168818    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-tls\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.168856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168854    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b81a300d-dfb4-4853-b354-e541e6c0aec4-config-volume\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168879    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59938fcc-4816-4ad1-9a47-413063935cb6-cert\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168920    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87615c5-a046-4c52-8d19-df38cbb27340-crio-socket\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168956    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-certificates\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.168992    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-image-registry-private-configuration\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169032    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklq7\" (UniqueName: \"kubernetes.io/projected/b81a300d-dfb4-4853-b354-e541e6c0aec4-kube-api-access-sklq7\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.169086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169034    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a87615c5-a046-4c52-8d19-df38cbb27340-crio-socket\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169111    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b81a300d-dfb4-4853-b354-e541e6c0aec4-tmp-dir\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169180    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6edeecf4-b1fb-42bf-8a2a-f0c379443841-tmp\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169220    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87615c5-a046-4c52-8d19-df38cbb27340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169262    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-ca-trust-extracted\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169290    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-installation-pull-secrets\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169330    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87615c5-a046-4c52-8d19-df38cbb27340-data-volume\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.169384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169370    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-trusted-ca\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169395    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpv5\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-kube-api-access-tbpv5\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.169759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169588    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6edeecf4-b1fb-42bf-8a2a-f0c379443841-tmp\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.169759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169754    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b81a300d-dfb4-4853-b354-e541e6c0aec4-config-volume\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.169925 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169755    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a87615c5-a046-4c52-8d19-df38cbb27340-data-volume\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.169981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.169931    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a87615c5-a046-4c52-8d19-df38cbb27340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.171086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.171067    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a87615c5-a046-4c52-8d19-df38cbb27340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.171192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.171138    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b81a300d-dfb4-4853-b354-e541e6c0aec4-metrics-tls\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.171346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.171329    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59938fcc-4816-4ad1-9a47-413063935cb6-cert\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.189706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.189666    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklq7\" (UniqueName: \"kubernetes.io/projected/b81a300d-dfb4-4853-b354-e541e6c0aec4-kube-api-access-sklq7\") pod \"dns-default-n2nn6\" (UID: \"b81a300d-dfb4-4853-b354-e541e6c0aec4\") " pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.189968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.189945    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntb7\" (UniqueName: \"kubernetes.io/projected/59938fcc-4816-4ad1-9a47-413063935cb6-kube-api-access-pntb7\") pod \"ingress-canary-89vxf\" (UID: \"59938fcc-4816-4ad1-9a47-413063935cb6\") " pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.190447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.190421    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52rc\" (UniqueName: \"kubernetes.io/projected/6edeecf4-b1fb-42bf-8a2a-f0c379443841-kube-api-access-z52rc\") pod \"downloads-6dd789c8b7-529p7\" (UID: \"6edeecf4-b1fb-42bf-8a2a-f0c379443841\") " pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.193329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.193309    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5kh\" (UniqueName: \"kubernetes.io/projected/a87615c5-a046-4c52-8d19-df38cbb27340-kube-api-access-5r5kh\") pod \"insights-runtime-extractor-cvd84\" (UID: \"a87615c5-a046-4c52-8d19-df38cbb27340\") " pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.198755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.198735    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:24.264801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.264781    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:24.269718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269692    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-tls\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.269820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269734    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-certificates\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.269820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269762    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-image-registry-private-configuration\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.269820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269790    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-ca-trust-extracted\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.269820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269805    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-installation-pull-secrets\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.269824    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-trusted-ca\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270150    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpv5\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-kube-api-access-tbpv5\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270222    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-bound-sa-token\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270265    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-ca-trust-extracted\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270627    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-certificates\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.270734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270686    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:24.270836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.270787    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-trusted-ca\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.272411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.272385    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-image-registry-private-configuration\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.273115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.272993    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-installation-pull-secrets\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.273115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.273082    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-registry-tls\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.290635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.289429    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-bound-sa-token\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.290635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.289503    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpv5\" (UniqueName: \"kubernetes.io/projected/401dd476-ba95-4b49-9e3c-3c7ad7b386c4-kube-api-access-tbpv5\") pod \"image-registry-77469d6cc6-zjsth\" (UID: \"401dd476-ba95-4b49-9e3c-3c7ad7b386c4\") " pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.292403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.292210    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89vxf"
Oct 30 19:04:24.301571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.301548    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvd84"
Oct 30 19:04:24.362821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.362786    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:24.485099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.484953    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"]
Oct 30 19:04:24.489399 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.489375    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe020691_8510_4e02_9ee5_d01d497d6710.slice/crio-bc9157d77f2b4fa31c0cd20e2b625f979ef3232288e182132e26e101a4493c8c WatchSource:0}: Error finding container bc9157d77f2b4fa31c0cd20e2b625f979ef3232288e182132e26e101a4493c8c: Status 404 returned error can't find the container with id bc9157d77f2b4fa31c0cd20e2b625f979ef3232288e182132e26e101a4493c8c
Oct 30 19:04:24.565672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.565645    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2nn6"]
Oct 30 19:04:24.569609 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.569567    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81a300d_dfb4_4853_b354_e541e6c0aec4.slice/crio-d29131ae6a1fe707a2069a54d2ce6e19eb352028f9fbd13f1a575f393819f1e7 WatchSource:0}: Error finding container d29131ae6a1fe707a2069a54d2ce6e19eb352028f9fbd13f1a575f393819f1e7: Status 404 returned error can't find the container with id d29131ae6a1fe707a2069a54d2ce6e19eb352028f9fbd13f1a575f393819f1e7
Oct 30 19:04:24.582430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.582408    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-529p7"]
Oct 30 19:04:24.586400 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.586377    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edeecf4_b1fb_42bf_8a2a_f0c379443841.slice/crio-a3369d2df8dde4a6dc02e3279371ea1f78b1f0a40e3a5b138b28c066f139e9eb WatchSource:0}: Error finding container a3369d2df8dde4a6dc02e3279371ea1f78b1f0a40e3a5b138b28c066f139e9eb: Status 404 returned error can't find the container with id a3369d2df8dde4a6dc02e3279371ea1f78b1f0a40e3a5b138b28c066f139e9eb
Oct 30 19:04:24.617876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.617853    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89vxf"]
Oct 30 19:04:24.620964 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.620945    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59938fcc_4816_4ad1_9a47_413063935cb6.slice/crio-39d8a5d6de6c53c86cde9425af870a9867b8acde705dce4e6d75060d704dee72 WatchSource:0}: Error finding container 39d8a5d6de6c53c86cde9425af870a9867b8acde705dce4e6d75060d704dee72: Status 404 returned error can't find the container with id 39d8a5d6de6c53c86cde9425af870a9867b8acde705dce4e6d75060d704dee72
Oct 30 19:04:24.642381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.642340    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvd84"]
Oct 30 19:04:24.646344 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.646321    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87615c5_a046_4c52_8d19_df38cbb27340.slice/crio-5d81551eaa890790525c2d165cd42387f79662bbcf84d7d3e4e173c890e85d0d WatchSource:0}: Error finding container 5d81551eaa890790525c2d165cd42387f79662bbcf84d7d3e4e173c890e85d0d: Status 404 returned error can't find the container with id 5d81551eaa890790525c2d165cd42387f79662bbcf84d7d3e4e173c890e85d0d
Oct 30 19:04:24.668292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:24.668270    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77469d6cc6-zjsth"]
Oct 30 19:04:24.671420 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:24.671369    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401dd476_ba95_4b49_9e3c_3c7ad7b386c4.slice/crio-884d5f64e56b5d6f0688f43f58593eaee829fdd9bd09bd3f17a6c71da0f3f1c6 WatchSource:0}: Error finding container 884d5f64e56b5d6f0688f43f58593eaee829fdd9bd09bd3f17a6c71da0f3f1c6: Status 404 returned error can't find the container with id 884d5f64e56b5d6f0688f43f58593eaee829fdd9bd09bd3f17a6c71da0f3f1c6
Oct 30 19:04:25.158181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.158102    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-529p7" event={"ID":"6edeecf4-b1fb-42bf-8a2a-f0c379443841","Type":"ContainerStarted","Data":"a3369d2df8dde4a6dc02e3279371ea1f78b1f0a40e3a5b138b28c066f139e9eb"}
Oct 30 19:04:25.159834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.159807    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2nn6" event={"ID":"b81a300d-dfb4-4853-b354-e541e6c0aec4","Type":"ContainerStarted","Data":"d29131ae6a1fe707a2069a54d2ce6e19eb352028f9fbd13f1a575f393819f1e7"}
Oct 30 19:04:25.161520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.161495    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd" event={"ID":"be020691-8510-4e02-9ee5-d01d497d6710","Type":"ContainerStarted","Data":"bc9157d77f2b4fa31c0cd20e2b625f979ef3232288e182132e26e101a4493c8c"}
Oct 30 19:04:25.165153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.165124    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvd84" event={"ID":"a87615c5-a046-4c52-8d19-df38cbb27340","Type":"ContainerStarted","Data":"53ef30da7b4f1662cc3d66281c1891a3548bd6cd8966d180c61d1104dd941d1c"}
Oct 30 19:04:25.165261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.165183    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvd84" event={"ID":"a87615c5-a046-4c52-8d19-df38cbb27340","Type":"ContainerStarted","Data":"5d81551eaa890790525c2d165cd42387f79662bbcf84d7d3e4e173c890e85d0d"}
Oct 30 19:04:25.169089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.169063    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth" event={"ID":"401dd476-ba95-4b49-9e3c-3c7ad7b386c4","Type":"ContainerStarted","Data":"2b413a84006f18c8ce9ee12a865df88736422a1100e0adecf224f74d1db83c30"}
Oct 30 19:04:25.169188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.169105    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth" event={"ID":"401dd476-ba95-4b49-9e3c-3c7ad7b386c4","Type":"ContainerStarted","Data":"884d5f64e56b5d6f0688f43f58593eaee829fdd9bd09bd3f17a6c71da0f3f1c6"}
Oct 30 19:04:25.169188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.169147    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:25.171143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.171097    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89vxf" event={"ID":"59938fcc-4816-4ad1-9a47-413063935cb6","Type":"ContainerStarted","Data":"39d8a5d6de6c53c86cde9425af870a9867b8acde705dce4e6d75060d704dee72"}
Oct 30 19:04:25.209545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.209501    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth" podStartSLOduration=1.209488845 podStartE2EDuration="1.209488845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:25.209116621 +0000 UTC m=+13.244629096" watchObservedRunningTime="2026-10-30 19:04:25.209488845 +0000 UTC m=+13.245001309"
Oct 30 19:04:25.900759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.900731    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:04:25.902725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.902703    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.925677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.925557    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:04:25.925770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.925712    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 19:04:25.925867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.925852    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 19:04:25.926175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.926159    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xbz79\""
Oct 30 19:04:25.926253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.926240    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 19:04:25.926336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.926323    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 19:04:25.926415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.926401    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 19:04:25.985021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.984993    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985068    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985104    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985143    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985192    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppks\" (UniqueName: \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985263    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:25.985324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:25.985299    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086423    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086463    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086491    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086510    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086524    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppks\" (UniqueName: \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086550    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.086581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.086567    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.087320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.087288    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.087657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.087557    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.093074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.092738    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.093074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.093015    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.100741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.100705    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.109043 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.108970    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.112168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.112143    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppks\" (UniqueName: \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\") pod \"console-7f9b7bdc46-bj5p2\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") " pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.184628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.178981    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd" event={"ID":"be020691-8510-4e02-9ee5-d01d497d6710","Type":"ContainerStarted","Data":"90bb5d55f44cd6c80d39baa3cf174927e31a3d2aa09bb52c5f22dd6c754f09d5"}
Oct 30 19:04:26.184628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.179730    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:26.184628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.182843    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvd84" event={"ID":"a87615c5-a046-4c52-8d19-df38cbb27340","Type":"ContainerStarted","Data":"15cb528eea341864496e0b394068fb7d88cf660d1c3de2f7ef2920cf45d60737"}
Oct 30 19:04:26.186414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.186324    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd"
Oct 30 19:04:26.216726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.216704    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:26.220190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.220151    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-56cvd" podStartSLOduration=1.8703187959999998 podStartE2EDuration="3.220140483s" totalImagesPullingTime="1.349821687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:26.219584592 +0000 UTC m=+14.255097055" watchObservedRunningTime="2026-10-30 19:04:26.220140483 +0000 UTC m=+14.255652957"
Oct 30 19:04:26.562358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.562281    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:04:26.618916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.617811    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"]
Oct 30 19:04:26.620834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.620786    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.636344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636325    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:04:26.636607 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636576    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlj2j\""
Oct 30 19:04:26.636712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636685    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:04:26.636775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636746    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:04:26.636877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636840    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:04:26.636941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.636892    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:04:26.648469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.648437    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"]
Oct 30 19:04:26.693410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.693126    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjqb\" (UniqueName: \"kubernetes.io/projected/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-kube-api-access-rxjqb\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.693410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.693173    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.693410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.693239    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.693410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.693277    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.794534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.794501    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.794697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.794547    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.794697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.794577    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjqb\" (UniqueName: \"kubernetes.io/projected/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-kube-api-access-rxjqb\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.794697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.794617    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.796121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.796072    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.800302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.800278    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.800956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.800935    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.815412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.815356    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjqb\" (UniqueName: \"kubernetes.io/projected/d8b4f72b-b606-4aad-8f47-fc6ef7d146a5-kube-api-access-rxjqb\") pod \"prometheus-operator-9cd89b74c-qdnjd\" (UID: \"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:26.936109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:26.936076    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"
Oct 30 19:04:27.079859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:27.079719    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ckfxz_2357170c-bfb3-4ee9-925f-e84d351a2c86/dns-node-resolver/0.log"
Oct 30 19:04:27.191749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:27.191557    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2nn6" event={"ID":"b81a300d-dfb4-4853-b354-e541e6c0aec4","Type":"ContainerStarted","Data":"74bcb4a25f4c5443108a3e971197131f39fa82f0f644ea56e9febf5baa0154f8"}
Oct 30 19:04:27.195141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:27.195117    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9b7bdc46-bj5p2" event={"ID":"2803fb1d-f912-4414-a5f9-f05d246516c3","Type":"ContainerStarted","Data":"ca08c5084f8c22b4ffbf496b7ba1f358d959ae02ab15f0df7b75147bd4799d23"}
Oct 30 19:04:27.266520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:27.266484    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd"]
Oct 30 19:04:27.281863 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:27.281832    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b4f72b_b606_4aad_8f47_fc6ef7d146a5.slice/crio-7f7870c25da689169e74ac2bca1107a68935d1193cffc4f0174178e478a97351 WatchSource:0}: Error finding container 7f7870c25da689169e74ac2bca1107a68935d1193cffc4f0174178e478a97351: Status 404 returned error can't find the container with id 7f7870c25da689169e74ac2bca1107a68935d1193cffc4f0174178e478a97351
Oct 30 19:04:28.080935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.079939    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77469d6cc6-zjsth_401dd476-ba95-4b49-9e3c-3c7ad7b386c4/registry/0.log"
Oct 30 19:04:28.200787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.200751    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd" event={"ID":"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5","Type":"ContainerStarted","Data":"7f7870c25da689169e74ac2bca1107a68935d1193cffc4f0174178e478a97351"}
Oct 30 19:04:28.219777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.219741    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2nn6" event={"ID":"b81a300d-dfb4-4853-b354-e541e6c0aec4","Type":"ContainerStarted","Data":"f37beb6dbcc26cd8bdb37f834479c9fa344f68ddc9e4df8eed8b28e79ca133f0"}
Oct 30 19:04:28.220307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.220278    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:28.225319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.225297    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvd84" event={"ID":"a87615c5-a046-4c52-8d19-df38cbb27340","Type":"ContainerStarted","Data":"e557aa4da783be1d11729fa5da70b4ac064e9b8a5bcbdd691984962acb2a9ef3"}
Oct 30 19:04:28.258544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.258485    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-n2nn6" podStartSLOduration=2.812240566 podStartE2EDuration="5.258466916s" totalImagesPullingTime="2.44622635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:28.257673301 +0000 UTC m=+16.293185759" watchObservedRunningTime="2026-10-30 19:04:28.258466916 +0000 UTC m=+16.293979377"
Oct 30 19:04:28.279510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.279402    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wmz8_b56513e8-bfbe-4662-a549-6aaf87a53316/node-ca/0.log"
Oct 30 19:04:28.297494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:28.297443    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cvd84" podStartSLOduration=2.587022812 podStartE2EDuration="5.29743175s" totalImagesPullingTime="2.710408938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:28.296226931 +0000 UTC m=+16.331739396" watchObservedRunningTime="2026-10-30 19:04:28.29743175 +0000 UTC m=+16.332944211"
Oct 30 19:04:28.651787 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:28.651756    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:04:29.231212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:29.231176    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89vxf" event={"ID":"59938fcc-4816-4ad1-9a47-413063935cb6","Type":"ContainerStarted","Data":"56612e5fc22c53cf5fe83e08c40bf29bfc78563c00964bc2adb36a3673b63529"}
Oct 30 19:04:29.234592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:29.234567    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd" event={"ID":"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5","Type":"ContainerStarted","Data":"277021ebee9028ad6606302ac4247627b00b53b6e03ca2f6009450d4f2e46ac2"}
Oct 30 19:04:29.234720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:29.234632    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd" event={"ID":"d8b4f72b-b606-4aad-8f47-fc6ef7d146a5","Type":"ContainerStarted","Data":"0b8622830925fb90464da4a18ea47be8f8898c6c9432a1728c98681cf8806c3d"}
Oct 30 19:04:29.268954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:29.268910    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-89vxf" podStartSLOduration=2.301095531 podStartE2EDuration="6.268894039s" totalImagesPullingTime="3.967798508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:29.267650534 +0000 UTC m=+17.303163000" watchObservedRunningTime="2026-10-30 19:04:29.268894039 +0000 UTC m=+17.304406503"
Oct 30 19:04:29.305232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:29.305141    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qdnjd" podStartSLOduration=1.5684117880000001 podStartE2EDuration="3.305123574s" totalImagesPullingTime="1.736711786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:29.304469647 +0000 UTC m=+17.339982110" watchObservedRunningTime="2026-10-30 19:04:29.305123574 +0000 UTC m=+17.340636046"
Oct 30 19:04:31.194612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.193379    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"]
Oct 30 19:04:31.195665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.195642    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.220853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.220828    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:04:31.224119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.224098    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qrpkn\""
Oct 30 19:04:31.224245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.224141    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:04:31.225240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.225142    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"]
Oct 30 19:04:31.244423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.243075    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hg6xt"]
Oct 30 19:04:31.245572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.245551    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"]
Oct 30 19:04:31.245858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.245713    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.248090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.248069    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9b7bdc46-bj5p2" event={"ID":"2803fb1d-f912-4414-a5f9-f05d246516c3","Type":"ContainerStarted","Data":"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"}
Oct 30 19:04:31.248334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.248288    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.262915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.262891    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:04:31.263001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.262988    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:04:31.268471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.268449    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"]
Oct 30 19:04:31.282857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.282835    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:04:31.283054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.282879    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:04:31.283345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.283054    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sz8sw\""
Oct 30 19:04:31.283345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.283123    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qxgnc\""
Oct 30 19:04:31.283345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.283182    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:04:31.283345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.283237    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:04:31.344614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344568    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.344775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344632    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78011371-2b97-4cbd-b99b-bc29a7138874-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.344775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344712    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-wtmp\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.344881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344776    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-metrics-client-ca\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.344881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344842    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-tls\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.344977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.344916    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.345321 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345304    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-sys\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.345374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345361    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-root\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.345432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345415    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.345491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345455    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f04f850-94c5-409b-b58a-856349d6683c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.345792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345774    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rpn\" (UniqueName: \"kubernetes.io/projected/3f04f850-94c5-409b-b58a-856349d6683c-kube-api-access-j9rpn\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.345859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345815    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.345907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345889    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-textfile\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.345958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345910    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.345958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345934    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpxb\" (UniqueName: \"kubernetes.io/projected/78011371-2b97-4cbd-b99b-bc29a7138874-kube-api-access-4zpxb\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.346062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.345957    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.346062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.346046    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.346160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.346084    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.346160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.346120    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfhz\" (UniqueName: \"kubernetes.io/projected/42185bd3-6753-4e3b-a38e-5b172172e005-kube-api-access-5qfhz\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447074    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78011371-2b97-4cbd-b99b-bc29a7138874-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.447131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447104    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-wtmp\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447121    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-metrics-client-ca\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447189    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-tls\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447230    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447260    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-sys\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447279    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-wtmp\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447291    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-root\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447342    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447378    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f04f850-94c5-409b-b58a-856349d6683c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447405    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rpn\" (UniqueName: \"kubernetes.io/projected/3f04f850-94c5-409b-b58a-856349d6683c-kube-api-access-j9rpn\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447433    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447469    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-textfile\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447498    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447516    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpxb\" (UniqueName: \"kubernetes.io/projected/78011371-2b97-4cbd-b99b-bc29a7138874-kube-api-access-4zpxb\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447533    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447552    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447571    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.447618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447586    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfhz\" (UniqueName: \"kubernetes.io/projected/42185bd3-6753-4e3b-a38e-5b172172e005-kube-api-access-5qfhz\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447626    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447770    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-metrics-client-ca\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447858    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78011371-2b97-4cbd-b99b-bc29a7138874-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.447874    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-accelerators-collector-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.448004    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-textfile\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.448163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.448132    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-sys\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.448459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.448200    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42185bd3-6753-4e3b-a38e-5b172172e005-root\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.449466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.448937    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.449466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.449000    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f04f850-94c5-409b-b58a-856349d6683c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.449466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.449211    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f04f850-94c5-409b-b58a-856349d6683c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.450620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.450582    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-tls\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.450708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.450661    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.451387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.451361    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42185bd3-6753-4e3b-a38e-5b172172e005-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.451476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.451464    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.451704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.451677    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78011371-2b97-4cbd-b99b-bc29a7138874-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.451809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.451758    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f04f850-94c5-409b-b58a-856349d6683c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.467046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.467019    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfhz\" (UniqueName: \"kubernetes.io/projected/42185bd3-6753-4e3b-a38e-5b172172e005-kube-api-access-5qfhz\") pod \"node-exporter-hg6xt\" (UID: \"42185bd3-6753-4e3b-a38e-5b172172e005\") " pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.467911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.467889    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpxb\" (UniqueName: \"kubernetes.io/projected/78011371-2b97-4cbd-b99b-bc29a7138874-kube-api-access-4zpxb\") pod \"openshift-state-metrics-7bd7489957-58tfl\" (UID: \"78011371-2b97-4cbd-b99b-bc29a7138874\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.468171 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.468152    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rpn\" (UniqueName: \"kubernetes.io/projected/3f04f850-94c5-409b-b58a-856349d6683c-kube-api-access-j9rpn\") pod \"kube-state-metrics-5dc8d7d89b-v6nx4\" (UID: \"3f04f850-94c5-409b-b58a-856349d6683c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.509734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.509709    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"
Oct 30 19:04:31.564358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.564159    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hg6xt"
Oct 30 19:04:31.569918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.569889    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"
Oct 30 19:04:31.821609 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.820770    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f9b7bdc46-bj5p2" podStartSLOduration=3.1467655 podStartE2EDuration="6.820735456s" totalImagesPullingTime="3.673969956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:31.371156776 +0000 UTC m=+19.406669239" watchObservedRunningTime="2026-10-30 19:04:31.820735456 +0000 UTC m=+19.856247924"
Oct 30 19:04:31.821609 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.821476    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl"]
Oct 30 19:04:31.824906 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:31.824880    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78011371_2b97_4cbd_b99b_bc29a7138874.slice/crio-ddf229cdf849a0ef97d8c59c3928faf4218f33ac7325ffbd2284a85767a7d39b WatchSource:0}: Error finding container ddf229cdf849a0ef97d8c59c3928faf4218f33ac7325ffbd2284a85767a7d39b: Status 404 returned error can't find the container with id ddf229cdf849a0ef97d8c59c3928faf4218f33ac7325ffbd2284a85767a7d39b
Oct 30 19:04:31.877806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:31.877784    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4"]
Oct 30 19:04:31.882538 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:31.882508    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f04f850_94c5_409b_b58a_856349d6683c.slice/crio-44ea3a3ee8f8db7d02d271f1b4d3579ca931700c1f703ab818d05b0ecc62992d WatchSource:0}: Error finding container 44ea3a3ee8f8db7d02d271f1b4d3579ca931700c1f703ab818d05b0ecc62992d: Status 404 returned error can't find the container with id 44ea3a3ee8f8db7d02d271f1b4d3579ca931700c1f703ab818d05b0ecc62992d
Oct 30 19:04:32.250520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.250481    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4" event={"ID":"3f04f850-94c5-409b-b58a-856349d6683c","Type":"ContainerStarted","Data":"44ea3a3ee8f8db7d02d271f1b4d3579ca931700c1f703ab818d05b0ecc62992d"}
Oct 30 19:04:32.251933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.251903    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6xt" event={"ID":"42185bd3-6753-4e3b-a38e-5b172172e005","Type":"ContainerStarted","Data":"5f4c4a0f70a8a98d59c636b7359d240371fd3613d2141727bef68a4cfefcc8bc"}
Oct 30 19:04:32.254219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.254191    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl" event={"ID":"78011371-2b97-4cbd-b99b-bc29a7138874","Type":"ContainerStarted","Data":"4792040dc7eb16164d417a263a19b134c718d63ef730d422724cce5663fc5f00"}
Oct 30 19:04:32.254328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.254250    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl" event={"ID":"78011371-2b97-4cbd-b99b-bc29a7138874","Type":"ContainerStarted","Data":"27308c95bd51143ab3bf4f5d84faa58d20a5f1a4c829c72de6dc4f1f7063aaf9"}
Oct 30 19:04:32.254328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.254281    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl" event={"ID":"78011371-2b97-4cbd-b99b-bc29a7138874","Type":"ContainerStarted","Data":"ddf229cdf849a0ef97d8c59c3928faf4218f33ac7325ffbd2284a85767a7d39b"}
Oct 30 19:04:32.604041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.603962    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:04:32.608508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.608476    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.625707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.625680    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:04:32.625838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.625761    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:04:32.625896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.625838    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:04:32.626008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.625983    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:04:32.626066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.626005    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z7mb7\""
Oct 30 19:04:32.626066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.626009    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:04:32.626626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.626592    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:04:32.630876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.630856    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:04:32.633032 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.633011    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:04:32.646121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.646099    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:04:32.646220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.646100    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:04:32.763581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763544    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763619    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763654    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763684    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763709    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763748    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763796    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763826    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763851    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763875    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhds4\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763910    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763936    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.763973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.763971    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865197    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865232    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865267    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865297    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865335    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865369    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865397    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865421    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865455    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865513    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865550    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865576    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.865894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.865633    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhds4\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.867571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.867194    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.867571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.867245    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.867571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.867254    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.869830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.869648    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.871135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.870780    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.871135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.871087    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.871135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.871106    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.872511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.872456    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.872511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.872456    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.876000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.875957    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.876000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.875975    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.877527 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.877505    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.884954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.884930    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhds4\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\") pod \"alertmanager-main-0\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:32.934681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:32.934652    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:04:33.260515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.260481    4320 generic.go:356] "Generic (PLEG): container finished" podID="42185bd3-6753-4e3b-a38e-5b172172e005" containerID="72b0b8aa99167abbef1ca96ea47d40440ef93eba9564eabf0c122ac7cbc01668" exitCode=0
Oct 30 19:04:33.260972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.260585    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6xt" event={"ID":"42185bd3-6753-4e3b-a38e-5b172172e005","Type":"ContainerDied","Data":"72b0b8aa99167abbef1ca96ea47d40440ef93eba9564eabf0c122ac7cbc01668"}
Oct 30 19:04:33.264479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.264449    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl" event={"ID":"78011371-2b97-4cbd-b99b-bc29a7138874","Type":"ContainerStarted","Data":"cb00609f4339164572c8e783e7a85823cd25ff3dc600e9e743b1fb69caed36c9"}
Oct 30 19:04:33.274297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.274276    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:04:33.298337 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:33.298312    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34eb2f5b_7376_49ea_999d_433ae044356c.slice/crio-ef9ecee8c7f91b56978f6d3c70be2dc15039766c68d52a42cf229aab3e685e70 WatchSource:0}: Error finding container ef9ecee8c7f91b56978f6d3c70be2dc15039766c68d52a42cf229aab3e685e70: Status 404 returned error can't find the container with id ef9ecee8c7f91b56978f6d3c70be2dc15039766c68d52a42cf229aab3e685e70
Oct 30 19:04:33.401222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.401167    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-58tfl" podStartSLOduration=1.207476399 podStartE2EDuration="2.40114853s" totalImagesPullingTime="1.193672131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:33.342519343 +0000 UTC m=+21.378031806" watchObservedRunningTime="2026-10-30 19:04:33.40114853 +0000 UTC m=+21.436660995"
Oct 30 19:04:33.401395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.401379    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d49bbb759-7xwlv"]
Oct 30 19:04:33.406524 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.406503    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.424641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.424514    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:04:33.424641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.424530    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:04:33.424641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.424623    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:04:33.424795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.424628    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kcdm\""
Oct 30 19:04:33.424921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.424900    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7tg5o0mguf2l3\""
Oct 30 19:04:33.427156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.427050    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:04:33.427156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.427104    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:04:33.430030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.429947    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d49bbb759-7xwlv"]
Oct 30 19:04:33.554284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.553156    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kgjxh"
Oct 30 19:04:33.572931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.572905    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.572941    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnst\" (UniqueName: \"kubernetes.io/projected/62f34307-2c45-4d42-8340-8d4d2d63e43e-kube-api-access-xsnst\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573035    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-grpc-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573073    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573104    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573132    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62f34307-2c45-4d42-8340-8d4d2d63e43e-tmp\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573262    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573336    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.573410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.573393    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f34307-2c45-4d42-8340-8d4d2d63e43e-metrics-client-ca\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674257    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674308    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f34307-2c45-4d42-8340-8d4d2d63e43e-metrics-client-ca\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674333    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674355    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnst\" (UniqueName: \"kubernetes.io/projected/62f34307-2c45-4d42-8340-8d4d2d63e43e-kube-api-access-xsnst\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674374    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-grpc-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674403    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674421    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674438    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62f34307-2c45-4d42-8340-8d4d2d63e43e-tmp\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674477    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.674983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.674962    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62f34307-2c45-4d42-8340-8d4d2d63e43e-tmp\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.675570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.675546    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f34307-2c45-4d42-8340-8d4d2d63e43e-metrics-client-ca\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.677417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.677296    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.677488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.677411    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.678167 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.678120    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.678448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.678404    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.679149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.679102    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.680286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.680237    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62f34307-2c45-4d42-8340-8d4d2d63e43e-secret-grpc-tls\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.693345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.693316    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnst\" (UniqueName: \"kubernetes.io/projected/62f34307-2c45-4d42-8340-8d4d2d63e43e-kube-api-access-xsnst\") pod \"thanos-querier-d49bbb759-7xwlv\" (UID: \"62f34307-2c45-4d42-8340-8d4d2d63e43e\") " pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:33.740256 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:33.740226    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:34.082790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.082717    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d49bbb759-7xwlv"]
Oct 30 19:04:34.085788 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:34.085739    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f34307_2c45_4d42_8340_8d4d2d63e43e.slice/crio-21dcaeeeee28fb7bb126d47e53e93d48577e37ef350b2292255e5574829aa89e WatchSource:0}: Error finding container 21dcaeeeee28fb7bb126d47e53e93d48577e37ef350b2292255e5574829aa89e: Status 404 returned error can't find the container with id 21dcaeeeee28fb7bb126d47e53e93d48577e37ef350b2292255e5574829aa89e
Oct 30 19:04:34.270465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.270428    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"ef9ecee8c7f91b56978f6d3c70be2dc15039766c68d52a42cf229aab3e685e70"}
Oct 30 19:04:34.273154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.273120    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4" event={"ID":"3f04f850-94c5-409b-b58a-856349d6683c","Type":"ContainerStarted","Data":"6e0e7f998ade3183009867d07d640ce6386a3a4a35bec1e14bbf5791e08ead34"}
Oct 30 19:04:34.273293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.273183    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4" event={"ID":"3f04f850-94c5-409b-b58a-856349d6683c","Type":"ContainerStarted","Data":"89dd3ce4862a14bfcb5cc30d4ab1a0bd9e8d6b0508a42125ad50778082858632"}
Oct 30 19:04:34.273293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.273205    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4" event={"ID":"3f04f850-94c5-409b-b58a-856349d6683c","Type":"ContainerStarted","Data":"081f380e325914113e3c601f3b43c18ab1f4a18569e911f9bcdf30cb1d086b14"}
Oct 30 19:04:34.275716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.275689    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6xt" event={"ID":"42185bd3-6753-4e3b-a38e-5b172172e005","Type":"ContainerStarted","Data":"2b9e3e713592c355ec8b27227dfedd18187a5226f16aa6ab9981a7323706ac95"}
Oct 30 19:04:34.275843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.275741    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hg6xt" event={"ID":"42185bd3-6753-4e3b-a38e-5b172172e005","Type":"ContainerStarted","Data":"dada82886baa0a91b9b9f6682b642572c102f9378f07f5aa8793704d4bafc3e3"}
Oct 30 19:04:34.277151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.277129    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"21dcaeeeee28fb7bb126d47e53e93d48577e37ef350b2292255e5574829aa89e"}
Oct 30 19:04:34.314173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.314126    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v6nx4" podStartSLOduration=1.703823741 podStartE2EDuration="3.31411323s" totalImagesPullingTime="1.610289489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.312989454 +0000 UTC m=+22.348501918" watchObservedRunningTime="2026-10-30 19:04:34.31411323 +0000 UTC m=+22.349625693"
Oct 30 19:04:34.353066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:34.352980    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hg6xt" podStartSLOduration=2.187853511 podStartE2EDuration="3.352963277s" totalImagesPullingTime="1.083941862s" totalInitContainerRuntime="81.167904ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.351924128 +0000 UTC m=+22.387436582" watchObservedRunningTime="2026-10-30 19:04:34.352963277 +0000 UTC m=+22.388475742"
Oct 30 19:04:35.289301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.289262    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e" exitCode=0
Oct 30 19:04:35.291119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.291011    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e"}
Oct 30 19:04:35.704544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.704508    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57c6bff-q9z2d"]
Oct 30 19:04:35.708450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.708421    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.723445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.723376    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:04:35.723592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.723470    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:04:35.723592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.723539    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-911na51eqp4m3\""
Oct 30 19:04:35.735308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.735286    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:04:35.735308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.735300    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-97qxz\""
Oct 30 19:04:35.735494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.735297    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:04:35.736639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.736617    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c6bff-q9z2d"]
Oct 30 19:04:35.796080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796030    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqbj\" (UniqueName: \"kubernetes.io/projected/03034025-6350-4677-976e-17f832f5ee08-kube-api-access-bnqbj\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796110    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-tls\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796153    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-metrics-server-audit-profiles\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796181    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-client-certs\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796209    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796269    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03034025-6350-4677-976e-17f832f5ee08-audit-log\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.796369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.796309    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-client-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897344    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqbj\" (UniqueName: \"kubernetes.io/projected/03034025-6350-4677-976e-17f832f5ee08-kube-api-access-bnqbj\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897394    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-tls\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897421    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-metrics-server-audit-profiles\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897436    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-client-certs\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897457    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897513    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03034025-6350-4677-976e-17f832f5ee08-audit-log\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.897543    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-client-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.898713    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.899466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.898931    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03034025-6350-4677-976e-17f832f5ee08-metrics-server-audit-profiles\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.900389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.900349    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03034025-6350-4677-976e-17f832f5ee08-audit-log\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.901392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.901369    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-client-ca-bundle\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.902952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.902930    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-tls\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.903195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.903175    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03034025-6350-4677-976e-17f832f5ee08-secret-metrics-server-client-certs\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:35.919547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:35.919524    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqbj\" (UniqueName: \"kubernetes.io/projected/03034025-6350-4677-976e-17f832f5ee08-kube-api-access-bnqbj\") pod \"metrics-server-57c6bff-q9z2d\" (UID: \"03034025-6350-4677-976e-17f832f5ee08\") " pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:36.022903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.022815    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:36.070555 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.070523    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"]
Oct 30 19:04:36.075273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.075244    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.091807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.091663    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lv5sd\""
Oct 30 19:04:36.092103 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.091832    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:04:36.099689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.099641    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tv8\" (UniqueName: \"kubernetes.io/projected/e86a99a6-dcd2-429d-b35f-3da6b10661ba-kube-api-access-w9tv8\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.099689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.099682    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e86a99a6-dcd2-429d-b35f-3da6b10661ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.101562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.100865    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"]
Oct 30 19:04:36.200438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.200410    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tv8\" (UniqueName: \"kubernetes.io/projected/e86a99a6-dcd2-429d-b35f-3da6b10661ba-kube-api-access-w9tv8\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.200619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.200448    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e86a99a6-dcd2-429d-b35f-3da6b10661ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.203470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.203423    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e86a99a6-dcd2-429d-b35f-3da6b10661ba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.217085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.217066    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:36.217180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.217121    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:36.220138 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.220119    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tv8\" (UniqueName: \"kubernetes.io/projected/e86a99a6-dcd2-429d-b35f-3da6b10661ba-kube-api-access-w9tv8\") pod \"monitoring-plugin-77f4f76594-hvf4z\" (UID: \"e86a99a6-dcd2-429d-b35f-3da6b10661ba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.223906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.223886    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:36.303507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.303041    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:04:36.358989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.358923    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c6bff-q9z2d"]
Oct 30 19:04:36.365582 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:36.365551    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03034025_6350_4677_976e_17f832f5ee08.slice/crio-9c4bb7340dd3380966272ca7a8c14679ab90b44b63b9a768420113fbddcf934e WatchSource:0}: Error finding container 9c4bb7340dd3380966272ca7a8c14679ab90b44b63b9a768420113fbddcf934e: Status 404 returned error can't find the container with id 9c4bb7340dd3380966272ca7a8c14679ab90b44b63b9a768420113fbddcf934e
Oct 30 19:04:36.404749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.404723    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:36.717873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:36.717778    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"]
Oct 30 19:04:36.728559 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:36.728489    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86a99a6_dcd2_429d_b35f_3da6b10661ba.slice/crio-93d8d74aca9c944d7110003d0cacff0a9862195e32ca3d750b9c41fd34573478 WatchSource:0}: Error finding container 93d8d74aca9c944d7110003d0cacff0a9862195e32ca3d750b9c41fd34573478: Status 404 returned error can't find the container with id 93d8d74aca9c944d7110003d0cacff0a9862195e32ca3d750b9c41fd34573478
Oct 30 19:04:37.137099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.135905    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:04:37.140060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.139246    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.186932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.186902    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 19:04:37.193730 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.193697    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:04:37.213154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213122    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213168    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213209    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213254    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213278    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213311    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213350    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.213523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.213372    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pq75\" (UniqueName: \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.303536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.303504    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"29eda73a898712a82a06b3181cc2fb15b06732a6598263c48cf12f907621dcb3"}
Oct 30 19:04:37.303941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.303585    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"abe43c6610765741308fcc808cc1b6d63c29d71a6d647fcd8064b25b82a3a75e"}
Oct 30 19:04:37.303941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.303643    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"d8bbe98e4be08217fbecd6c96c1ab29d6885922ad3759af1c2615502dd324199"}
Oct 30 19:04:37.305506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.305415    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z" event={"ID":"e86a99a6-dcd2-429d-b35f-3da6b10661ba","Type":"ContainerStarted","Data":"93d8d74aca9c944d7110003d0cacff0a9862195e32ca3d750b9c41fd34573478"}
Oct 30 19:04:37.307429 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.307364    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d" event={"ID":"03034025-6350-4677-976e-17f832f5ee08","Type":"ContainerStarted","Data":"9c4bb7340dd3380966272ca7a8c14679ab90b44b63b9a768420113fbddcf934e"}
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.313980    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314009    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314033    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314057    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314070    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314088    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314111    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314125    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pq75\" (UniqueName: \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.314968    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.316063    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.316212    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.316967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.316730    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.317474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.317444    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.318023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.317987    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.320962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.320941    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.335337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.335311    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pq75\" (UniqueName: \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\") pod \"console-6dfbcd7846-rcp6f\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") " pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.455124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.454343    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:37.773287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:37.773255    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:04:37.784228 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:37.784203    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361d6b4e_bced_47fa_b989_7d791fe7d401.slice/crio-cb385299963b72c57ca3dbae381960e03aa52ffb0c6aa3c8231091e9da942249 WatchSource:0}: Error finding container cb385299963b72c57ca3dbae381960e03aa52ffb0c6aa3c8231091e9da942249: Status 404 returned error can't find the container with id cb385299963b72c57ca3dbae381960e03aa52ffb0c6aa3c8231091e9da942249
Oct 30 19:04:38.222366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.222303    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:04:38.229267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.227848    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.240139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.240117    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2nn6"
Oct 30 19:04:38.254119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254017    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:04:38.254119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254072    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:04:38.254302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254141    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:04:38.254359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254317    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:04:38.254650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254427    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:04:38.254650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254477    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:04:38.254650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254505    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:04:38.254650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254608    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:04:38.254906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.254665    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:04:38.255493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.255328    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:04:38.255493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.255463    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:04:38.255865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.255767    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8dqsj\""
Oct 30 19:04:38.256337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.256159    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-35sdhqm2c7eha\""
Oct 30 19:04:38.267637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.267560    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:04:38.275560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.275535    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:04:38.276261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.276237    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:04:38.277895 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.277828    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:04:38.325486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325458    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.325486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325492    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325516    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325582    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325620    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325644    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325680    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325699    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325744    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325775    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325807    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325833    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325848    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325903    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.325925    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326203    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326452    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326614    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326660    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.326916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326814    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"}
Oct 30 19:04:38.326916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326859    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqqh4\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.327013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.326957    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"}
Oct 30 19:04:38.327062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.327021    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"}
Oct 30 19:04:38.327112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.327077    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"}
Oct 30 19:04:38.327161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.327126    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"}
Oct 30 19:04:38.327209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.327183    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerStarted","Data":"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"}
Oct 30 19:04:38.330204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.330176    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcd7846-rcp6f" event={"ID":"361d6b4e-bced-47fa-b989-7d791fe7d401","Type":"ContainerStarted","Data":"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"}
Oct 30 19:04:38.330317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.330223    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcd7846-rcp6f" event={"ID":"361d6b4e-bced-47fa-b989-7d791fe7d401","Type":"ContainerStarted","Data":"cb385299963b72c57ca3dbae381960e03aa52ffb0c6aa3c8231091e9da942249"}
Oct 30 19:04:38.335250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.335225    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"6b989a631ade5eb7e339055ca500455042328e8a65fd5d33f2db69edb68ddf00"}
Oct 30 19:04:38.335357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.335301    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"8aa4ed58b7618878771dc7ebbb5785f49c9fb699d1a7622d505b39b035c415b8"}
Oct 30 19:04:38.335357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.335342    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" event={"ID":"62f34307-2c45-4d42-8340-8d4d2d63e43e","Type":"ContainerStarted","Data":"264784a4a495b4ef5a16b7a867642b4285998ec8b1725b28e8fadd016de88ef3"}
Oct 30 19:04:38.335791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.335753    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:38.428002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.427976    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428028    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428138    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428163    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428235    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428320    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428345    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428397    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428419    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428444    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428473    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428500    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428549    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428652    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428679    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428796    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqqh4\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428842    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428870    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.428942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.428902    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.429148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.429025    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.430159 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.430134    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.430362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.430337    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.431034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.431011    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.431493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.431469    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.432739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.432709    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.433785    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.435807    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.437532    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.438265    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.438790    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.438807    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.439296    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.439661    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.439870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.439780    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.440379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.440333    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.440908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.440557    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.440908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.440575    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.440908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.440754    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.441123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.441044    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.389763609 podStartE2EDuration="6.441030543s" totalImagesPullingTime="3.037113403s" totalInitContainerRuntime="14.153531ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:38.434242582 +0000 UTC m=+26.469755060" watchObservedRunningTime="2026-10-30 19:04:38.441030543 +0000 UTC m=+26.476543012"
Oct 30 19:04:38.444094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.444056    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.456119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.455996    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqqh4\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\") pod \"prometheus-k8s-0\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.522774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.520967    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6dfbcd7846-rcp6f" podStartSLOduration=1.5209499260000001 podStartE2EDuration="1.520949926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:38.476015236 +0000 UTC m=+26.511527715" watchObservedRunningTime="2026-10-30 19:04:38.520949926 +0000 UTC m=+26.556462392"
Oct 30 19:04:38.522774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.522158    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv" podStartSLOduration=2.254115768 podStartE2EDuration="5.522146105s" totalImagesPullingTime="3.268030337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:38.519893654 +0000 UTC m=+26.555406122" watchObservedRunningTime="2026-10-30 19:04:38.522146105 +0000 UTC m=+26.557658571"
Oct 30 19:04:38.564830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.564799    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:38.918553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:38.911160    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:04:38.918553 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:04:38.912542    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a05017a_9a3f_4d5b_97bf_2a642fe9a973.slice/crio-a39ef41ac1b18563ad97b0cafa17d9264f5d1562af55f7ab55e24e100764837e WatchSource:0}: Error finding container a39ef41ac1b18563ad97b0cafa17d9264f5d1562af55f7ab55e24e100764837e: Status 404 returned error can't find the container with id a39ef41ac1b18563ad97b0cafa17d9264f5d1562af55f7ab55e24e100764837e
Oct 30 19:04:39.025037 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:39.025001    4320 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a05017a_9a3f_4d5b_97bf_2a642fe9a973.slice/crio-3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:04:39.342680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.342639    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d" event={"ID":"03034025-6350-4677-976e-17f832f5ee08","Type":"ContainerStarted","Data":"5782cf7b2c78e8073e3876688450b329c0d18b7546337f65d9644e6ff0dd5f30"}
Oct 30 19:04:39.344852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.344818    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z" event={"ID":"e86a99a6-dcd2-429d-b35f-3da6b10661ba","Type":"ContainerStarted","Data":"5698b2b8ca1b771a86712a3ea630db484443add5b2674c5a7143f4ffdd16fdff"}
Oct 30 19:04:39.345002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.344964    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:39.346984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.346956    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f" exitCode=0
Oct 30 19:04:39.347111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.347041    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"}
Oct 30 19:04:39.347174 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.347112    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"a39ef41ac1b18563ad97b0cafa17d9264f5d1562af55f7ab55e24e100764837e"}
Oct 30 19:04:39.351308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.351289    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z"
Oct 30 19:04:39.421516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.421463    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d" podStartSLOduration=2.400412089 podStartE2EDuration="4.421449503s" totalImagesPullingTime="2.021037414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:39.384046644 +0000 UTC m=+27.419559102" watchObservedRunningTime="2026-10-30 19:04:39.421449503 +0000 UTC m=+27.456961967"
Oct 30 19:04:39.422048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:39.421976    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hvf4z" podStartSLOduration=1.286254736 podStartE2EDuration="3.421963893s" totalImagesPullingTime="2.135709157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:39.42054941 +0000 UTC m=+27.456061873" watchObservedRunningTime="2026-10-30 19:04:39.421963893 +0000 UTC m=+27.457476359"
Oct 30 19:04:42.275705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:42.275673    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rrwhk"
Oct 30 19:04:44.130051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.129989    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r5zck"
Oct 30 19:04:44.355091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.355061    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d49bbb759-7xwlv"
Oct 30 19:04:44.374786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374753    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"}
Oct 30 19:04:44.374932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374831    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"}
Oct 30 19:04:44.374932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374863    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"}
Oct 30 19:04:44.374932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374890    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"}
Oct 30 19:04:44.374932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374927    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"}
Oct 30 19:04:44.375091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.374974    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerStarted","Data":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"}
Oct 30 19:04:44.450008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:44.449945    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.521323245 podStartE2EDuration="6.449925845s" totalImagesPullingTime="3.913726216s" totalInitContainerRuntime="14.876384ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:44.448789078 +0000 UTC m=+32.484301545" watchObservedRunningTime="2026-10-30 19:04:44.449925845 +0000 UTC m=+32.485438310"
Oct 30 19:04:46.189944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:46.189915    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77469d6cc6-zjsth"
Oct 30 19:04:46.386638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:46.386576    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-529p7" event={"ID":"6edeecf4-b1fb-42bf-8a2a-f0c379443841","Type":"ContainerStarted","Data":"de2dc1461e3c5b7b998f2bd24462f610f7885534b7aaba21ca52df44f4f743fb"}
Oct 30 19:04:46.387360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:46.387333    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:46.406624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:46.406276    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-529p7"
Oct 30 19:04:46.432679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:46.432263    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-529p7" podStartSLOduration=2.5998002700000002 podStartE2EDuration="23.432245164s" totalImagesPullingTime="20.832444894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:46.431485937 +0000 UTC m=+34.466998423" watchObservedRunningTime="2026-10-30 19:04:46.432245164 +0000 UTC m=+34.467757629"
Oct 30 19:04:47.454974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:47.454939    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:47.455456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:47.455009    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:47.460798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:47.460769    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:48.400102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:48.400072    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:04:48.467122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:48.467085    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:04:48.565880 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:48.565843    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:04:56.023119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:56.023049    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:56.023119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:04:56.023103    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:04:58.643741 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:04:58.643704    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:05:11.395190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:11.395154    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:11.442820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:11.442791    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:11.500385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:11.500359    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:13.492729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.492662    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f9b7bdc46-bj5p2" podUID="2803fb1d-f912-4414-a5f9-f05d246516c3" containerName="console" containerID="cri-o://de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539" gracePeriod=15
Oct 30 19:05:13.741578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.741554    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:05:13.863076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863010    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863051    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863075    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863107    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863144    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863348    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\" (UniqueName: \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863385    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:13.863441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863433    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\") pod \"2803fb1d-f912-4414-a5f9-f05d246516c3\" (UID: \"2803fb1d-f912-4414-a5f9-f05d246516c3\") "
Oct 30 19:05:13.863569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863543    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:13.863644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863554    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:13.863644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863565    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:13.863810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863788    4320 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-oauth-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.863810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863808    4320 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.863993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863817    4320 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2803fb1d-f912-4414-a5f9-f05d246516c3-tmp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.863993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.863825    4320 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2803fb1d-f912-4414-a5f9-f05d246516c3-console-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.865459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.865430    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:13.865459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.865448    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:13.865576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.865504    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks" pod "2803fb1d-f912-4414-a5f9-f05d246516c3" (UID: "2803fb1d-f912-4414-a5f9-f05d246516c3"). InnerVolumeSpecName "kube-api-access-hppks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:13.965037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.965013    4320 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-oauth-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.965037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.965035    4320 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2803fb1d-f912-4414-a5f9-f05d246516c3-console-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:13.965160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:13.965044    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hppks\" (UniqueName: \"kubernetes.io/projected/2803fb1d-f912-4414-a5f9-f05d246516c3-kube-api-access-hppks\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:14.500716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.500684    4320 generic.go:356] "Generic (PLEG): container finished" podID="2803fb1d-f912-4414-a5f9-f05d246516c3" containerID="de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539" exitCode=0
Oct 30 19:05:14.501122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.500728    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9b7bdc46-bj5p2" event={"ID":"2803fb1d-f912-4414-a5f9-f05d246516c3","Type":"ContainerDied","Data":"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"}
Oct 30 19:05:14.501122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.500736    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9b7bdc46-bj5p2"
Oct 30 19:05:14.501122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.500779    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9b7bdc46-bj5p2" event={"ID":"2803fb1d-f912-4414-a5f9-f05d246516c3","Type":"ContainerDied","Data":"ca08c5084f8c22b4ffbf496b7ba1f358d959ae02ab15f0df7b75147bd4799d23"}
Oct 30 19:05:14.501122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.500817    4320 scope.go:122] "RemoveContainer" containerID="de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"
Oct 30 19:05:14.509132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.509112    4320 scope.go:122] "RemoveContainer" containerID="de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"
Oct 30 19:05:14.509396 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:14.509375    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539\": container with ID starting with de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539 not found: ID does not exist" containerID="de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"
Oct 30 19:05:14.509466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.509406    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539"} err="failed to get container status \"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539\": rpc error: code = NotFound desc = could not find container \"de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539\": container with ID starting with de6944677decdfbc1342baf88c840c8608f852f0c49355aa2a98ffc3f9130539 not found: ID does not exist"
Oct 30 19:05:14.550224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.550188    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:05:14.562073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:14.562053    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f9b7bdc46-bj5p2"]
Oct 30 19:05:16.028694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:16.028661    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:05:16.032539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:16.032517    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57c6bff-q9z2d"
Oct 30 19:05:16.093806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:16.093783    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2803fb1d-f912-4414-a5f9-f05d246516c3" path="/var/lib/kubelet/pods/2803fb1d-f912-4414-a5f9-f05d246516c3/volumes"
Oct 30 19:05:22.927892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.927859    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928237    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="alertmanager" containerID="cri-o://b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c" gracePeriod=120
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928285    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy" containerID="cri-o://c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d" gracePeriod=120
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928347    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="config-reloader" containerID="cri-o://a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7" gracePeriod=120
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928356    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-web" containerID="cri-o://df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2" gracePeriod=120
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928304    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-metric" containerID="cri-o://95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3" gracePeriod=120
Oct 30 19:05:22.928470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:22.928263    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="prom-label-proxy" containerID="cri-o://ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2" gracePeriod=120
Oct 30 19:05:23.541246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541215    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2" exitCode=0
Oct 30 19:05:23.541246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541242    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d" exitCode=0
Oct 30 19:05:23.541246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541249    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2" exitCode=0
Oct 30 19:05:23.541246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541258    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7" exitCode=0
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541266    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c" exitCode=0
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541289    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"}
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541393    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"}
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541423    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"}
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541448    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"}
Oct 30 19:05:23.541508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:23.541473    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"}
Oct 30 19:05:24.206826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.206805    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:05:24.245517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245463    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245499    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245522    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245540    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245574    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245638    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245671    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245695    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245732    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245765    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245811    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245852    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.245931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.245896    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\") pod \"34eb2f5b-7376-49ea-999d-433ae044356c\" (UID: \"34eb2f5b-7376-49ea-999d-433ae044356c\") "
Oct 30 19:05:24.246220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.246087    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:24.247351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.246338    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.247351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.246323    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-metrics-client-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.247351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.247282    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:24.248774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.248749    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.248884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.248858    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.248954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.248881    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.249013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.248984    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.249806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.249776    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:24.249889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.249776    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "kube-api-access-rhds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:24.250465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.250446    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.250834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.250816    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:24.257757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.257734    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.260055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.260035    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config" pod "34eb2f5b-7376-49ea-999d-433ae044356c" (UID: "34eb2f5b-7376-49ea-999d-433ae044356c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:24.346828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346800    4320 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346826    4320 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346837    4320 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-main-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346864    4320 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-web-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346874    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhds4\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-kube-api-access-rhds4\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346883    4320 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-cluster-tls-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346892    4320 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-main-db\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346901    4320 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34eb2f5b-7376-49ea-999d-433ae044356c-config-out\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346909    4320 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34eb2f5b-7376-49ea-999d-433ae044356c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346917    4320 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346927    4320 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/34eb2f5b-7376-49ea-999d-433ae044356c-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.346988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.346935    4320 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34eb2f5b-7376-49ea-999d-433ae044356c-tls-assets\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:24.548938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.548874    4320 generic.go:356] "Generic (PLEG): container finished" podID="34eb2f5b-7376-49ea-999d-433ae044356c" containerID="95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3" exitCode=0
Oct 30 19:05:24.549079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.548953    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"}
Oct 30 19:05:24.549079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.548969    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:05:24.549079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.549038    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"34eb2f5b-7376-49ea-999d-433ae044356c","Type":"ContainerDied","Data":"ef9ecee8c7f91b56978f6d3c70be2dc15039766c68d52a42cf229aab3e685e70"}
Oct 30 19:05:24.549229 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.549087    4320 scope.go:122] "RemoveContainer" containerID="ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"
Oct 30 19:05:24.557627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.557593    4320 scope.go:122] "RemoveContainer" containerID="95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"
Oct 30 19:05:24.564901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.564880    4320 scope.go:122] "RemoveContainer" containerID="c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"
Oct 30 19:05:24.571453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.571439    4320 scope.go:122] "RemoveContainer" containerID="df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"
Oct 30 19:05:24.577852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.577834    4320 scope.go:122] "RemoveContainer" containerID="a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"
Oct 30 19:05:24.593737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.593720    4320 scope.go:122] "RemoveContainer" containerID="b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"
Oct 30 19:05:24.600557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.600541    4320 scope.go:122] "RemoveContainer" containerID="48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e"
Oct 30 19:05:24.602422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.602401    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:05:24.607369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.607354    4320 scope.go:122] "RemoveContainer" containerID="ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"
Oct 30 19:05:24.607647 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.607629    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2\": container with ID starting with ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2 not found: ID does not exist" containerID="ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"
Oct 30 19:05:24.607700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.607656    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2"} err="failed to get container status \"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2\": rpc error: code = NotFound desc = could not find container \"ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2\": container with ID starting with ec5deb8a71cbd38231e513d4388e87cfa9a40937b70b56f79fcbf6cbfa4517e2 not found: ID does not exist"
Oct 30 19:05:24.607700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.607670    4320 scope.go:122] "RemoveContainer" containerID="95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"
Oct 30 19:05:24.607900 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.607886    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3\": container with ID starting with 95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3 not found: ID does not exist" containerID="95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"
Oct 30 19:05:24.607945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.607904    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3"} err="failed to get container status \"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3\": rpc error: code = NotFound desc = could not find container \"95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3\": container with ID starting with 95bcb4b2fe0abec506f34629c98c9537ac4c5c6aa535903ba366fe286adecbd3 not found: ID does not exist"
Oct 30 19:05:24.607945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.607915    4320 scope.go:122] "RemoveContainer" containerID="c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"
Oct 30 19:05:24.608119 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.608104    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d\": container with ID starting with c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d not found: ID does not exist" containerID="c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"
Oct 30 19:05:24.608158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608123    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d"} err="failed to get container status \"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d\": rpc error: code = NotFound desc = could not find container \"c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d\": container with ID starting with c0a7210324a8c3a266344e2d1d843a7d2fd87030233111bef8a9fc4f1083703d not found: ID does not exist"
Oct 30 19:05:24.608158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608137    4320 scope.go:122] "RemoveContainer" containerID="df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"
Oct 30 19:05:24.608331 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.608316    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2\": container with ID starting with df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2 not found: ID does not exist" containerID="df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"
Oct 30 19:05:24.608371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608334    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2"} err="failed to get container status \"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2\": rpc error: code = NotFound desc = could not find container \"df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2\": container with ID starting with df0f965b8a4f5de24bf8dbe60a9055f73ffe223aa11aadce79de9ca52f5be7f2 not found: ID does not exist"
Oct 30 19:05:24.608371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608347    4320 scope.go:122] "RemoveContainer" containerID="a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"
Oct 30 19:05:24.608544 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.608529    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7\": container with ID starting with a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7 not found: ID does not exist" containerID="a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"
Oct 30 19:05:24.608583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608547    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7"} err="failed to get container status \"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7\": rpc error: code = NotFound desc = could not find container \"a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7\": container with ID starting with a71965dbdc61b7df27b143ce5feea8ba3cddd1533bfe7e0bb5dd2ce8676f9fe7 not found: ID does not exist"
Oct 30 19:05:24.608583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608558    4320 scope.go:122] "RemoveContainer" containerID="b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"
Oct 30 19:05:24.608753 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.608739    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c\": container with ID starting with b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c not found: ID does not exist" containerID="b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"
Oct 30 19:05:24.608789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608756    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c"} err="failed to get container status \"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c\": rpc error: code = NotFound desc = could not find container \"b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c\": container with ID starting with b52affe770075c74e36642b5e672506e70585cad99d928e6e6d8a07d78c9439c not found: ID does not exist"
Oct 30 19:05:24.608789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608768    4320 scope.go:122] "RemoveContainer" containerID="48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e"
Oct 30 19:05:24.608981 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:24.608966    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e\": container with ID starting with 48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e not found: ID does not exist" containerID="48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e"
Oct 30 19:05:24.609029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.608984    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e"} err="failed to get container status \"48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e\": rpc error: code = NotFound desc = could not find container \"48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e\": container with ID starting with 48b5a7c5182cf3f007f01368f71f74f05259cea989724e36a608465d8228635e not found: ID does not exist"
Oct 30 19:05:24.614413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:24.614393    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:05:26.093680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:26.093643    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" path="/var/lib/kubelet/pods/34eb2f5b-7376-49ea-999d-433ae044356c/volumes"
Oct 30 19:05:27.516692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.516651    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:27.517130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517082    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="prometheus" containerID="cri-o://1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b" gracePeriod=600
Oct 30 19:05:27.517198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517112    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-web" containerID="cri-o://8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b" gracePeriod=600
Oct 30 19:05:27.517198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517186    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="config-reloader" containerID="cri-o://ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648" gracePeriod=600
Oct 30 19:05:27.517312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517114    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="thanos-sidecar" containerID="cri-o://4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe" gracePeriod=600
Oct 30 19:05:27.517312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517119    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37" gracePeriod=600
Oct 30 19:05:27.517312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.517284    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy" containerID="cri-o://fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be" gracePeriod=600
Oct 30 19:05:27.806773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.806746    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:27.876003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.875975    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876026    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876065    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876106    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876136    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876193    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876220    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876255    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876284    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876312    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876351    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876403    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876441    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876461    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876474    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876505    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876539    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876576    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876655    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876694    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.876738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876739    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\") pod \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\" (UID: \"8a05017a-9a3f-4d5b-97bf-2a642fe9a973\") "
Oct 30 19:05:27.877197 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.876836    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.877621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.877580    4320 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.877726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.877625    4320 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-metrics-client-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.878176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.878128    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.878256 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.878183    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.879492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.879464    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:27.880184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.880153    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.882002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.881975    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.882099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.882059    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.882181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.882160    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.882440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.882422    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.882777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.882744    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.883146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.883120    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.883277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.883259    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.883364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.883348    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "kube-api-access-jqqh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:27.884374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.884342    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:27.884463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.884396    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.885018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.884996    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.886156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.886127    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:27.886780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.886755    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:27.892417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.892396    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config" pod "8a05017a-9a3f-4d5b-97bf-2a642fe9a973" (UID: "8a05017a-9a3f-4d5b-97bf-2a642fe9a973"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:27.978001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.977974    4320 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.977998    4320 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978008    4320 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-db\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978017    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqqh4\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-kube-api-access-jqqh4\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978026    4320 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-kube-rbac-proxy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978038    4320 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-metrics-client-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978050    4320 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config-out\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978058    4320 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-grpc-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978067    4320 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978075    4320 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978085    4320 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978093    4320 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978102    4320 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978110    4320 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978120    4320 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-web-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978127    4320 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978136    4320 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:27.978153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:27.978144    4320 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a05017a-9a3f-4d5b-97bf-2a642fe9a973-tls-assets\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:05:28.569187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569154    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37" exitCode=0
Oct 30 19:05:28.569187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569183    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be" exitCode=0
Oct 30 19:05:28.569187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569191    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b" exitCode=0
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569199    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe" exitCode=0
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569208    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648" exitCode=0
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569215    4320 generic.go:356] "Generic (PLEG): container finished" podID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b" exitCode=0
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569236    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569250    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569310    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569341    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569366    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569378    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569397    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569428    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"}
Oct 30 19:05:28.569693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.569453    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a05017a-9a3f-4d5b-97bf-2a642fe9a973","Type":"ContainerDied","Data":"a39ef41ac1b18563ad97b0cafa17d9264f5d1562af55f7ab55e24e100764837e"}
Oct 30 19:05:28.578732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.578639    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.585661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.585643    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.592079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.592059    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.598626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.598589    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.604855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.604837    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.611623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.611608    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.618019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.617989    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.618307 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.618279    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.618386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618309    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.618386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618331    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.618607 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.618579    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.618708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618691    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.618753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618709    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.618967 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.618949    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.619035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618975    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.619035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.618996    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.619229 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.619212    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.619267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619234    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.619267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619248    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.619466 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.619450    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.619517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619472    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.619517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619485    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.619735 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.619720    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.619780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619739    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.619780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.619751    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.620020 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.620001    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.620100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620023    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.620100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620043    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.620275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620258    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.620275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620274    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.620523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620502    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.620523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620521    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.620750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620732    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.620750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620748    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.620955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620937    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.620955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.620954    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.621179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621161    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.621179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621177    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.621376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621353    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.621376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621374    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.621582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621563    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.621582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621582    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.621816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621798    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.621816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621814    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.621991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621974    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.621991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.621991    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.622186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622166    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.622247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622186    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.622396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622377    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.622396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622395    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.622573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622557    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.622662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622572    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.622783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622766    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.622783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622782    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.622967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622950    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.623037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.622966    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.623140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623125    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.623140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623138    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.623340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623322    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.623405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623340    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.623544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623517    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.623649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623543    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.623801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623780    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.623801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.623799    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.624021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624002    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.624080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624022    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.624247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624230    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.624301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624246    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.624460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624442    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.624518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624461    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.624707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624688    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.624707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624706    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.624948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624926    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.624997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.624950    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.625167 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625146    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.625235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625168    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.625395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625377    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.625443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625395    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.625617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625583    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.625617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625616    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.625829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625811    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.625910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.625832    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.626075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626050    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.626117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626078    4320 scope.go:122] "RemoveContainer" containerID="f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"
Oct 30 19:05:28.626276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626256    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37"} err="failed to get container status \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": rpc error: code = NotFound desc = could not find container \"f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37\": container with ID starting with f358b459a626cdc3fb8c6e1d25c9b4917a8b6a6b040607dddeec4bbcd5231c37 not found: ID does not exist"
Oct 30 19:05:28.626360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626278    4320 scope.go:122] "RemoveContainer" containerID="fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"
Oct 30 19:05:28.626511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626486    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be"} err="failed to get container status \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": rpc error: code = NotFound desc = could not find container \"fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be\": container with ID starting with fb2360bba094aea3a27ee90e448b3038c9976e7784e68eb5476ee2ded893b1be not found: ID does not exist"
Oct 30 19:05:28.626511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626510    4320 scope.go:122] "RemoveContainer" containerID="8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"
Oct 30 19:05:28.626850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626823    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b"} err="failed to get container status \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": rpc error: code = NotFound desc = could not find container \"8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b\": container with ID starting with 8fe0817cf1f12923d942ef5e6462e2af9a0abc464bdf4d0e5509490b1172b51b not found: ID does not exist"
Oct 30 19:05:28.626899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.626851    4320 scope.go:122] "RemoveContainer" containerID="4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"
Oct 30 19:05:28.627076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627058    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe"} err="failed to get container status \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": rpc error: code = NotFound desc = could not find container \"4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe\": container with ID starting with 4424273c7c8f2902d5a042d6a9664c55cf383b2fd3993d9e27dadce9b930bdbe not found: ID does not exist"
Oct 30 19:05:28.627076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627076    4320 scope.go:122] "RemoveContainer" containerID="ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"
Oct 30 19:05:28.627290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627271    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648"} err="failed to get container status \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": rpc error: code = NotFound desc = could not find container \"ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648\": container with ID starting with ab296d9f1e36a7d8a471ca21b6ac6d7b389c3f37901140c39ec59c1d4e0b5648 not found: ID does not exist"
Oct 30 19:05:28.627361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627302    4320 scope.go:122] "RemoveContainer" containerID="1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"
Oct 30 19:05:28.627514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627497    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b"} err="failed to get container status \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": rpc error: code = NotFound desc = could not find container \"1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b\": container with ID starting with 1869f2d5acab95e53110526ef55bafca6e8c3ecba146b2007197e1e1c581b45b not found: ID does not exist"
Oct 30 19:05:28.627561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627516    4320 scope.go:122] "RemoveContainer" containerID="3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"
Oct 30 19:05:28.627777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.627751    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f"} err="failed to get container status \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": rpc error: code = NotFound desc = could not find container \"3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f\": container with ID starting with 3bf4377ab94d0f5bb28ccc150ce0eefd27e84248f47686b14a009c3ed00e985f not found: ID does not exist"
Oct 30 19:05:28.635108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.635089    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:28.643645 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:28.643622    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:05:28.647194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.647176    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:28.666206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666184    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:28.666480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666464    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="config-reloader"
Oct 30 19:05:28.666520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666482    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="config-reloader"
Oct 30 19:05:28.666520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666500    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-metric"
Oct 30 19:05:28.666520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666505    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-metric"
Oct 30 19:05:28.666520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666514    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="alertmanager"
Oct 30 19:05:28.666520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666519    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="alertmanager"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666533    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2803fb1d-f912-4414-a5f9-f05d246516c3" containerName="console"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666538    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2803fb1d-f912-4414-a5f9-f05d246516c3" containerName="console"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666548    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666556    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666563    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666567    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666574    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="thanos-sidecar"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666579    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="thanos-sidecar"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666584    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-thanos"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666588    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-thanos"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666608    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="init-config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666614    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="init-config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666621    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="prometheus"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666627    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="prometheus"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666638    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666643    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666649    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666653    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666659    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="prom-label-proxy"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666663    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="prom-label-proxy"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666671    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="init-config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666675    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="init-config-reloader"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666681    4320 cpu_manager.go:414] "containerMap: removing container" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy"
Oct 30 19:05:28.666689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666685    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666742    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-metric"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666750    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="alertmanager"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666758    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2803fb1d-f912-4414-a5f9-f05d246516c3" containerName="console"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666762    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="config-reloader"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666768    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666774    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="thanos-sidecar"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666780    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy-thanos"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666790    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="prometheus"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666797    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" containerName="kube-rbac-proxy"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666803    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy-web"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666808    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="prom-label-proxy"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666813    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="kube-rbac-proxy"
Oct 30 19:05:28.667337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.666818    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="34eb2f5b-7376-49ea-999d-433ae044356c" containerName="config-reloader"
Oct 30 19:05:28.696382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.696359    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.700569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.700547    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:28.716516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716494    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:05:28.716516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716506    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:05:28.716686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716518    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:05:28.716686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716520    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:05:28.716686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716521    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:05:28.716845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716747    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:05:28.716845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716813    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:05:28.716845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716836    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:05:28.717000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716895    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:05:28.717000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716896    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:05:28.717000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716906    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:05:28.717000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.716933    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:05:28.717149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.717038    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8dqsj\""
Oct 30 19:05:28.717206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.717172    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-35sdhqm2c7eha\""
Oct 30 19:05:28.729706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.729684    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:05:28.729778 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.729753    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:05:28.783218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783197    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783221    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783242    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783260    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783391    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783458    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783489    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783551    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783578    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783617    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783638    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783698    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783722    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783738    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9djs\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-kube-api-access-t9djs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783764    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783786    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783826    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783912    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.783947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783939    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.784049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.783956    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885047    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885073    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885094    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885114    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885131    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885154    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885208    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885248    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885274    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9djs\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-kube-api-access-t9djs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885309    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885354    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885397    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885429    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885458    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885485    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885511    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885546    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885576    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.885945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.885641    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.886644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.886526    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.886884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.886860    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.888122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888060    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888238    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888341    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888375    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888546    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.888875    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.889261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.889201    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.890207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.889843    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.890207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.890075    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.891516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.890893    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.891516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.891253    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.891516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.891455    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.893665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.891854    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.893665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.892648    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.893665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.892914    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e436948-1c32-4bd5-bb49-91f517366e1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.895921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.894683    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e436948-1c32-4bd5-bb49-91f517366e1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.896522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.896504    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e436948-1c32-4bd5-bb49-91f517366e1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:28.908641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:28.908617    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9djs\" (UniqueName: \"kubernetes.io/projected/8e436948-1c32-4bd5-bb49-91f517366e1c-kube-api-access-t9djs\") pod \"prometheus-k8s-0\" (UID: \"8e436948-1c32-4bd5-bb49-91f517366e1c\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:29.115867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.115844    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:29.461967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.461898    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:05:29.465245 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:05:29.465217    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e436948_1c32_4bd5_bb49_91f517366e1c.slice/crio-2b66f8352eddf2d04787440abac64abb90a04167750d4cb1c79ec2f5de810cd8 WatchSource:0}: Error finding container 2b66f8352eddf2d04787440abac64abb90a04167750d4cb1c79ec2f5de810cd8: Status 404 returned error can't find the container with id 2b66f8352eddf2d04787440abac64abb90a04167750d4cb1c79ec2f5de810cd8
Oct 30 19:05:29.574923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.574898    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"f13509d7379246463e0da3c80f8e600c0186582b78ff2eeadf6c79b67a6e7b81"}
Oct 30 19:05:29.575319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.574977    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"2b66f8352eddf2d04787440abac64abb90a04167750d4cb1c79ec2f5de810cd8"}
Oct 30 19:05:29.883839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.883809    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:05:29.907888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.907866    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.910306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.910247    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:05:29.994877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.994845    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.994998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.994895    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qzb\" (UniqueName: \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.994993    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.995076    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.995101    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.995174    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995229 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.995192    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:29.995229 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:29.995210    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.094747 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.094722    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a05017a-9a3f-4d5b-97bf-2a642fe9a973" path="/var/lib/kubelet/pods/8a05017a-9a3f-4d5b-97bf-2a642fe9a973/volumes"
Oct 30 19:05:30.096266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096244    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096283    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096301    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096342    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096358    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096375    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096395    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.096740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.096415    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qzb\" (UniqueName: \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.097162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.097135    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.097285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.097137    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.097285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.097270    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.097406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.097331    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.097465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.097432    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.098754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.098729    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.098754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.098749    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.114975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.114934    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qzb\" (UniqueName: \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\") pod \"console-7865f5b96c-g4jrv\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") " pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.242202 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.242145    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:30.571287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.571169    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:05:30.574570 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:05:30.574541    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08317f7_48b7_4cff_ac75_3c38fcb8120c.slice/crio-3bf4bd6f7f2447a63a6541f19275fe7ef2fef097637a87a2f43465317165e14f WatchSource:0}: Error finding container 3bf4bd6f7f2447a63a6541f19275fe7ef2fef097637a87a2f43465317165e14f: Status 404 returned error can't find the container with id 3bf4bd6f7f2447a63a6541f19275fe7ef2fef097637a87a2f43465317165e14f
Oct 30 19:05:30.581301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.581272    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865f5b96c-g4jrv" event={"ID":"e08317f7-48b7-4cff-ac75-3c38fcb8120c","Type":"ContainerStarted","Data":"3bf4bd6f7f2447a63a6541f19275fe7ef2fef097637a87a2f43465317165e14f"}
Oct 30 19:05:30.582720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.582694    4320 generic.go:356] "Generic (PLEG): container finished" podID="8e436948-1c32-4bd5-bb49-91f517366e1c" containerID="f13509d7379246463e0da3c80f8e600c0186582b78ff2eeadf6c79b67a6e7b81" exitCode=0
Oct 30 19:05:30.582788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:30.582746    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerDied","Data":"f13509d7379246463e0da3c80f8e600c0186582b78ff2eeadf6c79b67a6e7b81"}
Oct 30 19:05:31.590407 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590372    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"72d6fb82ca3c7ebd10d8332e04c2c09c9e2893f71773a1c936fee7326cac784e"}
Oct 30 19:05:31.590888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590474    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"0d8cb7f54b69ff48d23756b1b1f71868a5cbeed808cc0a159ec227590dc66135"}
Oct 30 19:05:31.590888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590503    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"4cb0b7fc3c0cd010fada46caa97f70532479deebaa4e84910bd515c5077a0a1a"}
Oct 30 19:05:31.590888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590535    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"0556a3fda728aff85c3ff746a9f04d17669e0abbddde2b6647b2e5a88c7badf9"}
Oct 30 19:05:31.590888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590559    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"53940ff8f1d5e0900ba963013e126eb311f405bd78daff9085a7f129ee60decf"}
Oct 30 19:05:31.590888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.590584    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e436948-1c32-4bd5-bb49-91f517366e1c","Type":"ContainerStarted","Data":"47f91af0c630544839698cbc7c1775d23342d46d2feeefe6196679b99176d75b"}
Oct 30 19:05:31.591949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.591918    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865f5b96c-g4jrv" event={"ID":"e08317f7-48b7-4cff-ac75-3c38fcb8120c","Type":"ContainerStarted","Data":"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"}
Oct 30 19:05:31.650807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.650763    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.627146921 podStartE2EDuration="3.650750347s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.603426ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.648843197 +0000 UTC m=+79.684355661" watchObservedRunningTime="2026-10-30 19:05:31.650750347 +0000 UTC m=+79.686262810"
Oct 30 19:05:31.688774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:31.688731    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7865f5b96c-g4jrv" podStartSLOduration=2.688717599 podStartE2EDuration="2.688717599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.687243886 +0000 UTC m=+79.722756350" watchObservedRunningTime="2026-10-30 19:05:31.688717599 +0000 UTC m=+79.724230062"
Oct 30 19:05:34.116738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:34.116708    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:05:40.242434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:40.242391    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:40.243009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:40.242472    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:40.249143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:40.249112    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:40.635219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:40.635201    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:05:40.702745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:05:40.702716    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:05:58.643451 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:05:58.643415    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:06:05.727272 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:05.727194    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6dfbcd7846-rcp6f" podUID="361d6b4e-bced-47fa-b989-7d791fe7d401" containerName="console" containerID="cri-o://c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759" gracePeriod=15
Oct 30 19:06:05.975962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:05.975941    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:06:06.069253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069175    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\" (UniqueName: \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069228    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069273    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069343    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069372    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\" (UniqueName: \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069418    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069489    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069528    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\") pod \"361d6b4e-bced-47fa-b989-7d791fe7d401\" (UID: \"361d6b4e-bced-47fa-b989-7d791fe7d401\") "
Oct 30 19:06:06.069789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069715    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:06.069851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069840    4320 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/361d6b4e-bced-47fa-b989-7d791fe7d401-tmp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.069908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069730    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:06.069908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069889    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:06.070015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.069976    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:06.070084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.070058    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:06.071478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.071451    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:06.071478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.071465    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "kube-api-access-9pq75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:06.071648 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.071575    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert" pod "361d6b4e-bced-47fa-b989-7d791fe7d401" (UID: "361d6b4e-bced-47fa-b989-7d791fe7d401"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:06.170458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170435    4320 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-trusted-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170455    4320 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-oauth-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170464    4320 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170473    4320 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170483    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pq75\" (UniqueName: \"kubernetes.io/projected/361d6b4e-bced-47fa-b989-7d791fe7d401-kube-api-access-9pq75\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170492    4320 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/361d6b4e-bced-47fa-b989-7d791fe7d401-console-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.170665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.170500    4320 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/361d6b4e-bced-47fa-b989-7d791fe7d401-console-oauth-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:06:06.748323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.748291    4320 generic.go:356] "Generic (PLEG): container finished" podID="361d6b4e-bced-47fa-b989-7d791fe7d401" containerID="c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759" exitCode=0
Oct 30 19:06:06.748748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.748338    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcd7846-rcp6f"
Oct 30 19:06:06.748748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.748373    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcd7846-rcp6f" event={"ID":"361d6b4e-bced-47fa-b989-7d791fe7d401","Type":"ContainerDied","Data":"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"}
Oct 30 19:06:06.748748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.748419    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcd7846-rcp6f" event={"ID":"361d6b4e-bced-47fa-b989-7d791fe7d401","Type":"ContainerDied","Data":"cb385299963b72c57ca3dbae381960e03aa52ffb0c6aa3c8231091e9da942249"}
Oct 30 19:06:06.748748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.748440    4320 scope.go:122] "RemoveContainer" containerID="c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"
Oct 30 19:06:06.756301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.756281    4320 scope.go:122] "RemoveContainer" containerID="c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"
Oct 30 19:06:06.756591 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:06:06.756572    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759\": container with ID starting with c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759 not found: ID does not exist" containerID="c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"
Oct 30 19:06:06.756683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.756616    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759"} err="failed to get container status \"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759\": rpc error: code = NotFound desc = could not find container \"c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759\": container with ID starting with c9c1d465d5b6ac78fb62f4dcdc4a1605d617bb99a9a81c2c4196e48fd746d759 not found: ID does not exist"
Oct 30 19:06:06.797979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.797955    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:06:06.809825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:06.809801    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dfbcd7846-rcp6f"]
Oct 30 19:06:08.092977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:08.092947    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361d6b4e-bced-47fa-b989-7d791fe7d401" path="/var/lib/kubelet/pods/361d6b4e-bced-47fa-b989-7d791fe7d401/volumes"
Oct 30 19:06:28.644810 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:06:28.644722    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:06:29.117036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:29.116957    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:29.132267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:29.132243    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:29.850359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:29.850333    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:06:41.637584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.637555    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-646cbbdc7f-wsw55"]
Oct 30 19:06:41.638056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.637893    4320 cpu_manager.go:414] "containerMap: removing container" podUID="361d6b4e-bced-47fa-b989-7d791fe7d401" containerName="console"
Oct 30 19:06:41.638056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.637906    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361d6b4e-bced-47fa-b989-7d791fe7d401" containerName="console"
Oct 30 19:06:41.638056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.637971    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="361d6b4e-bced-47fa-b989-7d791fe7d401" containerName="console"
Oct 30 19:06:41.639901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.639886    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.661192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.661171    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-646cbbdc7f-wsw55"]
Oct 30 19:06:41.726360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726320    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-service-ca\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726373    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-oauth-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726467    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-oauth-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726498    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07648395-fb08-4c8c-a550-d0d0dfb0f657-tmp\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726616    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-trusted-ca-bundle\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726645    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726679    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.726800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.726742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v598j\" (UniqueName: \"kubernetes.io/projected/07648395-fb08-4c8c-a550-d0d0dfb0f657-kube-api-access-v598j\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827638    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-oauth-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827664    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-oauth-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827682    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07648395-fb08-4c8c-a550-d0d0dfb0f657-tmp\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827723    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-trusted-ca-bundle\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827750    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827784    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.827972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.827813    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v598j\" (UniqueName: \"kubernetes.io/projected/07648395-fb08-4c8c-a550-d0d0dfb0f657-kube-api-access-v598j\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828109    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07648395-fb08-4c8c-a550-d0d0dfb0f657-tmp\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828137    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-service-ca\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828416    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-oauth-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828551    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828763 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828743    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-trusted-ca-bundle\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.828865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.828845    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07648395-fb08-4c8c-a550-d0d0dfb0f657-service-ca\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.830317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.830296    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-serving-cert\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.830406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.830352    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07648395-fb08-4c8c-a550-d0d0dfb0f657-console-oauth-config\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.846767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.846745    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v598j\" (UniqueName: \"kubernetes.io/projected/07648395-fb08-4c8c-a550-d0d0dfb0f657-kube-api-access-v598j\") pod \"console-646cbbdc7f-wsw55\" (UID: \"07648395-fb08-4c8c-a550-d0d0dfb0f657\") " pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:41.950091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:41.950041    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:42.244884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:42.244811    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-646cbbdc7f-wsw55"]
Oct 30 19:06:42.249547 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:06:42.249521    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07648395_fb08_4c8c_a550_d0d0dfb0f657.slice/crio-d6fae77b8058b8f778e8a5f29ffd9cd7b08e3d5c48df55280170c221ec0421ab WatchSource:0}: Error finding container d6fae77b8058b8f778e8a5f29ffd9cd7b08e3d5c48df55280170c221ec0421ab: Status 404 returned error can't find the container with id d6fae77b8058b8f778e8a5f29ffd9cd7b08e3d5c48df55280170c221ec0421ab
Oct 30 19:06:42.884912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:42.884879    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646cbbdc7f-wsw55" event={"ID":"07648395-fb08-4c8c-a550-d0d0dfb0f657","Type":"ContainerStarted","Data":"bded88afa90daf38c800ab53f4165f76ea76178b7f68f1057dccfb67cd95e952"}
Oct 30 19:06:42.885347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:42.884923    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-646cbbdc7f-wsw55" event={"ID":"07648395-fb08-4c8c-a550-d0d0dfb0f657","Type":"ContainerStarted","Data":"d6fae77b8058b8f778e8a5f29ffd9cd7b08e3d5c48df55280170c221ec0421ab"}
Oct 30 19:06:51.950412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:51.950374    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:51.950966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:51.950459    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:51.955421 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:51.955395    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:51.999560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:51.999510    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-646cbbdc7f-wsw55" podStartSLOduration=10.999496559 podStartE2EDuration="10.999496559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:42.936189749 +0000 UTC m=+150.971702212" watchObservedRunningTime="2026-10-30 19:06:51.999496559 +0000 UTC m=+160.035009024"
Oct 30 19:06:52.945947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:52.945918    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-646cbbdc7f-wsw55"
Oct 30 19:06:53.013085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:06:53.013053    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:06:58.645084 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:06:58.645047    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:07:18.033373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.033332    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7865f5b96c-g4jrv" podUID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" containerName="console" containerID="cri-o://0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1" gracePeriod=15
Oct 30 19:07:18.263889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.263866    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:07:18.411972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.411948    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\" (UniqueName: \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412042    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412067    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412124    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412139    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412165    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412183    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412203    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\") pod \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\" (UID: \"e08317f7-48b7-4cff-ac75-3c38fcb8120c\") "
Oct 30 19:07:18.412734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412471    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.412734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412528    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:18.412734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412713    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:18.412955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412746    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:18.412955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.412894    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:18.414184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.414161    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "kube-api-access-x6qzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:18.414261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.414206    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:18.414261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.414240    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert" pod "e08317f7-48b7-4cff-ac75-3c38fcb8120c" (UID: "e08317f7-48b7-4cff-ac75-3c38fcb8120c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:18.513406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513370    4320 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-oauth-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513403    4320 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-oauth-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513413    4320 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513422    4320 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-serving-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513430    4320 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e08317f7-48b7-4cff-ac75-3c38fcb8120c-tmp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513439    4320 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-trusted-ca-bundle\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513448    4320 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e08317f7-48b7-4cff-ac75-3c38fcb8120c-console-config\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:18.513566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:18.513456    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6qzb\" (UniqueName: \"kubernetes.io/projected/e08317f7-48b7-4cff-ac75-3c38fcb8120c-kube-api-access-x6qzb\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:07:19.044357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.044322    4320 generic.go:356] "Generic (PLEG): container finished" podID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" containerID="0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1" exitCode=0
Oct 30 19:07:19.044773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.044377    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7865f5b96c-g4jrv"
Oct 30 19:07:19.044773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.044378    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865f5b96c-g4jrv" event={"ID":"e08317f7-48b7-4cff-ac75-3c38fcb8120c","Type":"ContainerDied","Data":"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"}
Oct 30 19:07:19.044773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.044519    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7865f5b96c-g4jrv" event={"ID":"e08317f7-48b7-4cff-ac75-3c38fcb8120c","Type":"ContainerDied","Data":"3bf4bd6f7f2447a63a6541f19275fe7ef2fef097637a87a2f43465317165e14f"}
Oct 30 19:07:19.044773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.044559    4320 scope.go:122] "RemoveContainer" containerID="0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"
Oct 30 19:07:19.053704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.053685    4320 scope.go:122] "RemoveContainer" containerID="0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"
Oct 30 19:07:19.053987 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:07:19.053969    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1\": container with ID starting with 0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1 not found: ID does not exist" containerID="0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"
Oct 30 19:07:19.054033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.053995    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1"} err="failed to get container status \"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1\": rpc error: code = NotFound desc = could not find container \"0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1\": container with ID starting with 0f6ef264b956730aaff525ef0e32075724f7dbff1e05cf01de42edd3d552d0f1 not found: ID does not exist"
Oct 30 19:07:19.093780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.093754    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:07:19.105726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:19.105704    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7865f5b96c-g4jrv"]
Oct 30 19:07:20.093037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:07:20.093006    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" path="/var/lib/kubelet/pods/e08317f7-48b7-4cff-ac75-3c38fcb8120c/volumes"
Oct 30 19:07:28.646058 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:07:28.646014    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:07:58.642148 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:07:58.642068    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:08:28.645389 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:08:28.645351    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:08:58.646456 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:08:58.646408    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:09:28.644458 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:09:28.644379    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:09:58.644219 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:09:58.644180    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:10:28.643582 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:10:28.643545    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:10:58.643247 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:10:58.643167    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:11:14.322301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.322271    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"]
Oct 30 19:11:14.322676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.322563    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" containerName="console"
Oct 30 19:11:14.322676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.322573    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" containerName="console"
Oct 30 19:11:14.322676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.322664    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08317f7-48b7-4cff-ac75-3c38fcb8120c" containerName="console"
Oct 30 19:11:14.325786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.325770    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.366702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.366676    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:11:14.366928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.366907    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:11:14.367012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.366916    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:11:14.367233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.367213    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:11:14.367370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.367237    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2lq27\""
Oct 30 19:11:14.367370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.367289    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:11:14.373990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.373959    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"]
Oct 30 19:11:14.416731 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416704    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwpv\" (UniqueName: \"kubernetes.io/projected/85d88a98-d8df-4e5f-a0f1-c210590f6dad-kube-api-access-bbwpv\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.416827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416738    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.416827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416818    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.416944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416845    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.416944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416926    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.417042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.416984    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.517681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517658    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.517798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517708    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.517798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517778    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwpv\" (UniqueName: \"kubernetes.io/projected/85d88a98-d8df-4e5f-a0f1-c210590f6dad-kube-api-access-bbwpv\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.517934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517903    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.518051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517961    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.518051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.517988    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.518171 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.518134    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.520214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.520187    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d88a98-d8df-4e5f-a0f1-c210590f6dad-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.520315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.520290    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.520315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.520296    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.520315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.520306    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85d88a98-d8df-4e5f-a0f1-c210590f6dad-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.536774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.536750    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwpv\" (UniqueName: \"kubernetes.io/projected/85d88a98-d8df-4e5f-a0f1-c210590f6dad-kube-api-access-bbwpv\") pod \"opendatahub-operator-controller-manager-8589ff54bc-8jpmq\" (UID: \"85d88a98-d8df-4e5f-a0f1-c210590f6dad\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:14.641971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:14.641940    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:15.003457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:15.003387    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"]
Oct 30 19:11:15.006999 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:11:15.006974    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d88a98_d8df_4e5f_a0f1_c210590f6dad.slice/crio-40a187c67ecded632bb93dbe15e64077c91323120df3ed823d04f71c83167e56 WatchSource:0}: Error finding container 40a187c67ecded632bb93dbe15e64077c91323120df3ed823d04f71c83167e56: Status 404 returned error can't find the container with id 40a187c67ecded632bb93dbe15e64077c91323120df3ed823d04f71c83167e56
Oct 30 19:11:15.008582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:15.008566    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:11:15.926539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:15.926502    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq" event={"ID":"85d88a98-d8df-4e5f-a0f1-c210590f6dad","Type":"ContainerStarted","Data":"40a187c67ecded632bb93dbe15e64077c91323120df3ed823d04f71c83167e56"}
Oct 30 19:11:17.934315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:17.934284    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq" event={"ID":"85d88a98-d8df-4e5f-a0f1-c210590f6dad","Type":"ContainerStarted","Data":"cdc9acd6215d45694722303636a972fb1455c80fc8d7d7ac108078ee37ea089d"}
Oct 30 19:11:18.939288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:18.939250    4320 generic.go:356] "Generic (PLEG): container finished" podID="85d88a98-d8df-4e5f-a0f1-c210590f6dad" containerID="cdc9acd6215d45694722303636a972fb1455c80fc8d7d7ac108078ee37ea089d" exitCode=0
Oct 30 19:11:18.939767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:18.939324    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq" event={"ID":"85d88a98-d8df-4e5f-a0f1-c210590f6dad","Type":"ContainerDied","Data":"cdc9acd6215d45694722303636a972fb1455c80fc8d7d7ac108078ee37ea089d"}
Oct 30 19:11:19.945019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:19.944983    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq" event={"ID":"85d88a98-d8df-4e5f-a0f1-c210590f6dad","Type":"ContainerStarted","Data":"beac79890cf20265cf6dcf0acd0f3e520acb43009cea0b664e6a18925886df9d"}
Oct 30 19:11:19.945478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:19.945128    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:20.038331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:20.038277    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq" podStartSLOduration=2.5344214149999997 podStartE2EDuration="6.038261913s" totalImagesPullingTime="3.09842467s" totalInitContainerRuntime="405.415828ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:20.037442079 +0000 UTC m=+428.072954543" watchObservedRunningTime="2026-10-30 19:11:20.038261913 +0000 UTC m=+428.073774431"
Oct 30 19:11:28.642819 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:11:28.642783    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:11:30.950991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:30.950958    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-8jpmq"
Oct 30 19:11:31.660773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.660743    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"]
Oct 30 19:11:31.666407 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.666386    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.701400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.701374    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"]
Oct 30 19:11:31.756052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.756025    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.756170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.756061    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/379c0868-30fe-4f71-be41-4bbaf3faea2e-manager-config\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.756170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.756084    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.756170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.756159    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5k6l\" (UniqueName: \"kubernetes.io/projected/379c0868-30fe-4f71-be41-4bbaf3faea2e-kube-api-access-r5k6l\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.760571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760541    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:11:31.760571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760549    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:11:31.760728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760571    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:11:31.760728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760583    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:11:31.760728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760609    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-svszl\""
Oct 30 19:11:31.760728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.760629    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:11:31.856823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.856798    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.856913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.856836    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k6l\" (UniqueName: \"kubernetes.io/projected/379c0868-30fe-4f71-be41-4bbaf3faea2e-kube-api-access-r5k6l\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.856913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.856889    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.857024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.856914    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/379c0868-30fe-4f71-be41-4bbaf3faea2e-manager-config\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.857509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.857484    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/379c0868-30fe-4f71-be41-4bbaf3faea2e-manager-config\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.859226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.859206    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.859302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.859229    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/379c0868-30fe-4f71-be41-4bbaf3faea2e-metrics-cert\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.875837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.875814    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5k6l\" (UniqueName: \"kubernetes.io/projected/379c0868-30fe-4f71-be41-4bbaf3faea2e-kube-api-access-r5k6l\") pod \"lws-controller-manager-5c54b56ff8-qzdjf\" (UID: \"379c0868-30fe-4f71-be41-4bbaf3faea2e\") " pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:31.977664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:31.977615    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:32.279306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:32.279222    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"]
Oct 30 19:11:32.282074 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:11:32.282049    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c0868_30fe_4f71_be41_4bbaf3faea2e.slice/crio-354ecef3f56c5fbb08a89c77b9fd734f21c97dfa574d5be5fa44ce7b8e02ae5f WatchSource:0}: Error finding container 354ecef3f56c5fbb08a89c77b9fd734f21c97dfa574d5be5fa44ce7b8e02ae5f: Status 404 returned error can't find the container with id 354ecef3f56c5fbb08a89c77b9fd734f21c97dfa574d5be5fa44ce7b8e02ae5f
Oct 30 19:11:32.998118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:32.998083    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf" event={"ID":"379c0868-30fe-4f71-be41-4bbaf3faea2e","Type":"ContainerStarted","Data":"354ecef3f56c5fbb08a89c77b9fd734f21c97dfa574d5be5fa44ce7b8e02ae5f"}
Oct 30 19:11:36.013206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:36.013160    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf" event={"ID":"379c0868-30fe-4f71-be41-4bbaf3faea2e","Type":"ContainerStarted","Data":"8686c0a5a214d0d71f9174c6883f2db946789438e124874025fdb542915e2fd4"}
Oct 30 19:11:36.013686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:36.013278    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:36.056802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:36.056758    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf" podStartSLOduration=2.318630944 podStartE2EDuration="5.056744862s" totalImagesPullingTime="2.738113918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:36.055873762 +0000 UTC m=+444.091386217" watchObservedRunningTime="2026-10-30 19:11:36.056744862 +0000 UTC m=+444.092257378"
Oct 30 19:11:47.020071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:47.020028    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c54b56ff8-qzdjf"
Oct 30 19:11:49.799927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.799893    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"]
Oct 30 19:11:49.802746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.802722    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:49.821355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.821336    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-p5skj\""
Oct 30 19:11:49.834359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.834339    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"]
Oct 30 19:11:49.909889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.909868    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:49.909989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.909897    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv27c\" (UniqueName: \"kubernetes.io/projected/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-kube-api-access-tv27c\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:49.909989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.909933    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:49.910075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.910027    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:49.929743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:49.929724    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 19:11:50.011172 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011150    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.011276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011190    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.011276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011240    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.011276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011259    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv27c\" (UniqueName: \"kubernetes.io/projected/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-kube-api-access-tv27c\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.011639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011624    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.011714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.011699    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.012099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.012081    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.030689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.030666    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv27c\" (UniqueName: \"kubernetes.io/projected/61f6a874-9cc9-4ceb-a304-dab54cf6a8be-kube-api-access-tv27c\") pod \"kserve-module-controller-manager-5cfb77bf8c-7dx5m\" (UID: \"61f6a874-9cc9-4ceb-a304-dab54cf6a8be\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.114324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.114269    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:50.423563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:50.423528    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"]
Oct 30 19:11:50.427359 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:11:50.427331    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f6a874_9cc9_4ceb_a304_dab54cf6a8be.slice/crio-f4439a0735fc4feffa08501c014ea7f115ba76ec119f2fed0bbfe6ca9564ca1a WatchSource:0}: Error finding container f4439a0735fc4feffa08501c014ea7f115ba76ec119f2fed0bbfe6ca9564ca1a: Status 404 returned error can't find the container with id f4439a0735fc4feffa08501c014ea7f115ba76ec119f2fed0bbfe6ca9564ca1a
Oct 30 19:11:51.023683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.023651    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"]
Oct 30 19:11:51.026966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.026219    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.040993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.040965    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wvqc9\""
Oct 30 19:11:51.044113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.044092    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:11:51.047086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.047051    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:11:51.047086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.047057    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:11:51.053289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.053268    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"]
Oct 30 19:11:51.079794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.079763    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m" event={"ID":"61f6a874-9cc9-4ceb-a304-dab54cf6a8be","Type":"ContainerStarted","Data":"f4439a0735fc4feffa08501c014ea7f115ba76ec119f2fed0bbfe6ca9564ca1a"}
Oct 30 19:11:51.120095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120070    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120112    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120154    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120191    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszfn\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-kube-api-access-nszfn\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120275    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120308    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120340    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120368    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.120578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.120403    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221248    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221292    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221333    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221360    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszfn\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-kube-api-access-nszfn\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221409    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221434    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221453    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221477    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221509    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.221949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221927    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.222030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.221971    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.222030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.222016    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.222110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.222036    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.222110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.222095    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.224259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.224230    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.224518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.224499    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.241899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.241876    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.241987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.241948    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszfn\" (UniqueName: \"kubernetes.io/projected/7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca-kube-api-access-nszfn\") pod \"data-science-gateway-data-science-gateway-class-7556cf8c574v98c\" (UID: \"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.342830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.342759    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:51.662978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:51.662945    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"]
Oct 30 19:11:51.669056 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:11:51.669028    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f28ddd5_8e1c_44aa_bd4e_c78bec9644ca.slice/crio-0db38ae1df2f46d750f61dca2d536f94610e1d2cd57f4b665fa74d0ba54aeef9 WatchSource:0}: Error finding container 0db38ae1df2f46d750f61dca2d536f94610e1d2cd57f4b665fa74d0ba54aeef9: Status 404 returned error can't find the container with id 0db38ae1df2f46d750f61dca2d536f94610e1d2cd57f4b665fa74d0ba54aeef9
Oct 30 19:11:52.094323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:52.094254    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c" event={"ID":"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca","Type":"ContainerStarted","Data":"0db38ae1df2f46d750f61dca2d536f94610e1d2cd57f4b665fa74d0ba54aeef9"}
Oct 30 19:11:53.096886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:53.096807    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m" event={"ID":"61f6a874-9cc9-4ceb-a304-dab54cf6a8be","Type":"ContainerStarted","Data":"be78cbc7f13f485c93cd022febdd638882c38e23cf7d37651e78a9c85bc86192"}
Oct 30 19:11:53.097305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:53.096920    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:11:53.151139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:53.150345    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m" podStartSLOduration=1.820149827 podStartE2EDuration="4.150326645s" totalImagesPullingTime="2.330176818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:53.148433969 +0000 UTC m=+461.183946444" watchObservedRunningTime="2026-10-30 19:11:53.150326645 +0000 UTC m=+461.185839112"
Oct 30 19:11:55.047566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:55.047530    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:11:55.047868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:55.047626    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:11:55.047868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:55.047665    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:11:56.113123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:56.113087    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c" event={"ID":"7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca","Type":"ContainerStarted","Data":"007dcd101d614a84fa75cbb6655e4dbf801af568465b5a65d1474467b19a8bc4"}
Oct 30 19:11:56.343177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:56.343147    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:56.347579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:56.347555    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:56.391622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:56.391552    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c" podStartSLOduration=3.015843002 podStartE2EDuration="6.391539609s" totalImagesPullingTime="3.375696607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:56.178117834 +0000 UTC m=+464.213630298" watchObservedRunningTime="2026-10-30 19:11:56.391539609 +0000 UTC m=+464.427052072"
Oct 30 19:11:57.117159 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:57.117130    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:57.118168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:11:57.118149    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7556cf8c574v98c"
Oct 30 19:11:58.644691 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:11:58.644642    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:12:14.101825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:14.101796    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-7dx5m"
Oct 30 19:12:28.642836 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:12:28.642761    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:12:37.262889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.262856    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lhwvl"]
Oct 30 19:12:37.269530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.269503    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.285470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.285443    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lhwvl"]
Oct 30 19:12:37.295924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.295907    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pwp6c\""
Oct 30 19:12:37.298758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.298736    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:12:37.300983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.300962    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:12:37.304180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.304162    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.323713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.323692    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:12:37.332994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.332973    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9r29w\""
Oct 30 19:12:37.332994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.332980    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:12:37.348505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.348482    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m6f5w"]
Oct 30 19:12:37.351781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.351763    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.373535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.373512    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m6f5w"]
Oct 30 19:12:37.389955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.389932    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:12:37.390071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.390018    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-d4swj\""
Oct 30 19:12:37.390196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.390172    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwj4n\" (UniqueName: \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.390257 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.390219    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-cert\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.390349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.390331    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.390391 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.390361    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4wf\" (UniqueName: \"kubernetes.io/projected/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-kube-api-access-dp4wf\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.398837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.398819    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gqrwf"]
Oct 30 19:12:37.402086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.402069    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.419476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.419453    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:12:37.421482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.421450    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-g6667\""
Oct 30 19:12:37.421578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.421456    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:12:37.423204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.423185    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gqrwf"]
Oct 30 19:12:37.491276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491257    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnw5\" (UniqueName: \"kubernetes.io/projected/92948160-ac67-4192-ad60-83336fc1158f-kube-api-access-ctnw5\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.491377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491291    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-cert\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.491377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491306    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.491377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491338    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwj4n\" (UniqueName: \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.491499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491477    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-cert\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.491548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491533    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g48t\" (UniqueName: \"kubernetes.io/projected/41afb888-9cc2-486a-b5f7-5e6fa8863e06-kube-api-access-9g48t\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.491586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491569    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.491653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491588    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4wf\" (UniqueName: \"kubernetes.io/projected/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-kube-api-access-dp4wf\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.491689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.491647    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92948160-ac67-4192-ad60-83336fc1158f-tls-certs\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.493860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.493841    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-cert\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.493958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.493887    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.511128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.511109    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4wf\" (UniqueName: \"kubernetes.io/projected/3ec07eb2-2af5-4d2b-ae2d-d905c57085f8-kube-api-access-dp4wf\") pod \"kserve-controller-manager-776cc66d47-lhwvl\" (UID: \"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.511231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.511202    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwj4n\" (UniqueName: \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\") pod \"llmisvc-controller-manager-869d5dcc5d-sjv8b\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.583010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.582959    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:37.592247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.592223    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnw5\" (UniqueName: \"kubernetes.io/projected/92948160-ac67-4192-ad60-83336fc1158f-kube-api-access-ctnw5\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.592340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.592280    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-cert\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.592340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.592332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.592456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.592427    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g48t\" (UniqueName: \"kubernetes.io/projected/41afb888-9cc2-486a-b5f7-5e6fa8863e06-kube-api-access-9g48t\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.592512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.592481    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92948160-ac67-4192-ad60-83336fc1158f-tls-certs\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.594586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.594566    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-cert\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.594942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.594918    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92948160-ac67-4192-ad60-83336fc1158f-tls-certs\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.595046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.594991    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41afb888-9cc2-486a-b5f7-5e6fa8863e06-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.611716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.611689    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnw5\" (UniqueName: \"kubernetes.io/projected/92948160-ac67-4192-ad60-83336fc1158f-kube-api-access-ctnw5\") pod \"model-serving-api-75bd89fdfc-m6f5w\" (UID: \"92948160-ac67-4192-ad60-83336fc1158f\") " pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.611809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.611785    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g48t\" (UniqueName: \"kubernetes.io/projected/41afb888-9cc2-486a-b5f7-5e6fa8863e06-kube-api-access-9g48t\") pod \"odh-model-controller-f87fcf56c-gqrwf\" (UID: \"41afb888-9cc2-486a-b5f7-5e6fa8863e06\") " pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.615494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.615474    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:37.665058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.664821    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:37.714430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.714395    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:37.878141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.878088    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lhwvl"]
Oct 30 19:12:37.881085 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:12:37.881050    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec07eb2_2af5_4d2b_ae2d_d905c57085f8.slice/crio-76c6cff3050711f4d6324fc5b74ccfc2604ad62b84067d55b5262802f21f5e88 WatchSource:0}: Error finding container 76c6cff3050711f4d6324fc5b74ccfc2604ad62b84067d55b5262802f21f5e88: Status 404 returned error can't find the container with id 76c6cff3050711f4d6324fc5b74ccfc2604ad62b84067d55b5262802f21f5e88
Oct 30 19:12:37.918318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.918267    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:12:37.921444 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:12:37.921420    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b785e21_5e53_4147_8433_ab9381dfa0c1.slice/crio-1e658fcd5690fb2a934db4c7f34feed7b14f234955a40d869b7dd11f416de8aa WatchSource:0}: Error finding container 1e658fcd5690fb2a934db4c7f34feed7b14f234955a40d869b7dd11f416de8aa: Status 404 returned error can't find the container with id 1e658fcd5690fb2a934db4c7f34feed7b14f234955a40d869b7dd11f416de8aa
Oct 30 19:12:37.963467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:37.963445    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m6f5w"]
Oct 30 19:12:37.966086 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:12:37.966060    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92948160_ac67_4192_ad60_83336fc1158f.slice/crio-da6459eb9c14ed213ddaa9c59356107fcd72a0745b8e324c0f86ce7646c20686 WatchSource:0}: Error finding container da6459eb9c14ed213ddaa9c59356107fcd72a0745b8e324c0f86ce7646c20686: Status 404 returned error can't find the container with id da6459eb9c14ed213ddaa9c59356107fcd72a0745b8e324c0f86ce7646c20686
Oct 30 19:12:38.015041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:38.015017    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gqrwf"]
Oct 30 19:12:38.017119 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:12:38.017098    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41afb888_9cc2_486a_b5f7_5e6fa8863e06.slice/crio-4cb666812cf2897903249bc5ada1d11f669459113ebbb632b2b148e95cfb1f28 WatchSource:0}: Error finding container 4cb666812cf2897903249bc5ada1d11f669459113ebbb632b2b148e95cfb1f28: Status 404 returned error can't find the container with id 4cb666812cf2897903249bc5ada1d11f669459113ebbb632b2b148e95cfb1f28
Oct 30 19:12:38.279717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:38.279688    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf" event={"ID":"41afb888-9cc2-486a-b5f7-5e6fa8863e06","Type":"ContainerStarted","Data":"4cb666812cf2897903249bc5ada1d11f669459113ebbb632b2b148e95cfb1f28"}
Oct 30 19:12:38.281237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:38.281209    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" event={"ID":"5b785e21-5e53-4147-8433-ab9381dfa0c1","Type":"ContainerStarted","Data":"1e658fcd5690fb2a934db4c7f34feed7b14f234955a40d869b7dd11f416de8aa"}
Oct 30 19:12:38.282412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:38.282388    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w" event={"ID":"92948160-ac67-4192-ad60-83336fc1158f","Type":"ContainerStarted","Data":"da6459eb9c14ed213ddaa9c59356107fcd72a0745b8e324c0f86ce7646c20686"}
Oct 30 19:12:38.283557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:38.283532    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerStarted","Data":"76c6cff3050711f4d6324fc5b74ccfc2604ad62b84067d55b5262802f21f5e88"}
Oct 30 19:12:41.304858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.304758    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" event={"ID":"5b785e21-5e53-4147-8433-ab9381dfa0c1","Type":"ContainerStarted","Data":"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"}
Oct 30 19:12:41.304858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.304822    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:12:41.306896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.306862    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w" event={"ID":"92948160-ac67-4192-ad60-83336fc1158f","Type":"ContainerStarted","Data":"8b7b7775e0b0c52c82d9958e375d1e785f939c59ce7aa6677007a3d88a464d18"}
Oct 30 19:12:41.307037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.306971    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:41.308419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.308394    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerStarted","Data":"aefeaa5a2643d059f216eac47b494db85ca51e2933bd4c15bd44f3a8bee98133"}
Oct 30 19:12:41.308944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.308924    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:41.344529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.344469    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" podStartSLOduration=1.328402463 podStartE2EDuration="4.344454857s" totalImagesPullingTime="3.016052394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:41.342468675 +0000 UTC m=+509.377981140" watchObservedRunningTime="2026-10-30 19:12:41.344454857 +0000 UTC m=+509.379967322"
Oct 30 19:12:41.381052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.381010    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" podStartSLOduration=1.44975331 podStartE2EDuration="4.380997626s" totalImagesPullingTime="2.931244316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:41.379808776 +0000 UTC m=+509.415321244" watchObservedRunningTime="2026-10-30 19:12:41.380997626 +0000 UTC m=+509.416510113"
Oct 30 19:12:41.418822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:41.418781    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w" podStartSLOduration=1.796162465 podStartE2EDuration="4.4187723s" totalImagesPullingTime="2.622609835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:41.417533834 +0000 UTC m=+509.453046299" watchObservedRunningTime="2026-10-30 19:12:41.4187723 +0000 UTC m=+509.454284763"
Oct 30 19:12:42.313835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:42.313796    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf" event={"ID":"41afb888-9cc2-486a-b5f7-5e6fa8863e06","Type":"ContainerStarted","Data":"217820be4af86a747620b85ee1d49d1add762a77487cef509c7f97b5ddab319a"}
Oct 30 19:12:42.314313 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:42.314296    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:42.352938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:42.352883    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf" podStartSLOduration=1.937501036 podStartE2EDuration="5.352869039s" totalImagesPullingTime="3.415368003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:42.351325478 +0000 UTC m=+510.386837942" watchObservedRunningTime="2026-10-30 19:12:42.352869039 +0000 UTC m=+510.388381504"
Oct 30 19:12:51.352767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:51.352679    4320 generic.go:356] "Generic (PLEG): container finished" podID="3ec07eb2-2af5-4d2b-ae2d-d905c57085f8" containerID="aefeaa5a2643d059f216eac47b494db85ca51e2933bd4c15bd44f3a8bee98133" exitCode=1
Oct 30 19:12:51.353234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:51.352763    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerDied","Data":"aefeaa5a2643d059f216eac47b494db85ca51e2933bd4c15bd44f3a8bee98133"}
Oct 30 19:12:51.353234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:51.353214    4320 scope.go:122] "RemoveContainer" containerID="aefeaa5a2643d059f216eac47b494db85ca51e2933bd4c15bd44f3a8bee98133"
Oct 30 19:12:52.318363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:52.318336    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-m6f5w"
Oct 30 19:12:52.358374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:52.358349    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerStarted","Data":"637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083"}
Oct 30 19:12:52.358888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:52.358547    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:12:54.323302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:12:54.323271    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-gqrwf"
Oct 30 19:12:58.644137 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:12:58.644096    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:13:02.402496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:02.402457    4320 generic.go:356] "Generic (PLEG): container finished" podID="3ec07eb2-2af5-4d2b-ae2d-d905c57085f8" containerID="637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083" exitCode=1
Oct 30 19:13:02.402927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:02.402531    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerDied","Data":"637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083"}
Oct 30 19:13:02.402927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:02.402611    4320 scope.go:122] "RemoveContainer" containerID="aefeaa5a2643d059f216eac47b494db85ca51e2933bd4c15bd44f3a8bee98133"
Oct 30 19:13:02.403080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:02.402933    4320 scope.go:122] "RemoveContainer" containerID="637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083"
Oct 30 19:13:02.403211 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:13:02.403191    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-776cc66d47-lhwvl_opendatahub(3ec07eb2-2af5-4d2b-ae2d-d905c57085f8)\"" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" podUID="3ec07eb2-2af5-4d2b-ae2d-d905c57085f8"
Oct 30 19:13:07.583808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:07.583774    4320 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:13:07.584233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:07.584112    4320 scope.go:122] "RemoveContainer" containerID="637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083"
Oct 30 19:13:07.588974 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:13:07.588951    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-776cc66d47-lhwvl_opendatahub(3ec07eb2-2af5-4d2b-ae2d-d905c57085f8)\"" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" podUID="3ec07eb2-2af5-4d2b-ae2d-d905c57085f8"
Oct 30 19:13:12.316515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:12.316487    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:13:14.089680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:14.089649    4320 scope.go:122] "RemoveContainer" containerID="637db7d27b174647438cd2b32b2df636b745c0f45f88061304ff4f0862cce083"
Oct 30 19:13:14.456907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:14.456871    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl" event={"ID":"3ec07eb2-2af5-4d2b-ae2d-d905c57085f8","Type":"ContainerStarted","Data":"471724ec6a3ff1d3b4b056b29b5e0337bcbc101396dfcb433939854e04b26383"}
Oct 30 19:13:14.457100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:14.457083    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:13:28.643848 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:13:28.643814    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:13:45.466645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:13:45.466608    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lhwvl"
Oct 30 19:13:58.643970 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:13:58.643933    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:14:28.644334 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:14:28.644295    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:14:58.644591 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:14:58.644549    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:15:13.080881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.080836    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"]
Oct 30 19:15:13.084325 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.084307    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.099148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.099124    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"]
Oct 30 19:15:13.130078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.130055    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:15:13.130230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.130170    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:15:13.148794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.148768    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzmp6\""
Oct 30 19:15:13.157233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.157205    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnmc\" (UniqueName: \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.157349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.157315    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.258175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.258137    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.258354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.258225    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnmc\" (UniqueName: \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.258561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.258539    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.277931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.277904    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnmc\" (UniqueName: \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\") pod \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.397856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.397826    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:13.723502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.723472    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"]
Oct 30 19:15:13.726709 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:15:13.726683    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66b10f0_dafd_44fe_84bb_d38e98d47599.slice/crio-7aac52a423b2d85f96300241e555d955e6b720169a2fd628122c0fa74069a9b3 WatchSource:0}: Error finding container 7aac52a423b2d85f96300241e555d955e6b720169a2fd628122c0fa74069a9b3: Status 404 returned error can't find the container with id 7aac52a423b2d85f96300241e555d955e6b720169a2fd628122c0fa74069a9b3
Oct 30 19:15:13.977931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:13.977842    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5" event={"ID":"e66b10f0-dafd-44fe-84bb-d38e98d47599","Type":"ContainerStarted","Data":"7aac52a423b2d85f96300241e555d955e6b720169a2fd628122c0fa74069a9b3"}
Oct 30 19:15:15.325403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.325361    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"]
Oct 30 19:15:15.327429 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.327400    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"]
Oct 30 19:15:15.348360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.347697    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:15.362541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.362516    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.381013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.380983    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:15.482348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.482311    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.482559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.482476    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlfv\" (UniqueName: \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.583287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.583202    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.583453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.583325    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlfv\" (UniqueName: \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.583677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.583648    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.605123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.605065    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlfv\" (UniqueName: \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\") pod \"kuadrant-operator-controller-manager-6b76784d48-lbw9m\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:15.683246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:15.683212    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:16.028268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:16.028231    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:16.044484 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:15:16.044450    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c70801_a4d6_4502_8ec2_133dd5fb95da.slice/crio-fe99c0b3a7fe04239e5de01405a02f8e94345c9740d0076fa5b5f8a78e986f7f WatchSource:0}: Error finding container fe99c0b3a7fe04239e5de01405a02f8e94345c9740d0076fa5b5f8a78e986f7f: Status 404 returned error can't find the container with id fe99c0b3a7fe04239e5de01405a02f8e94345c9740d0076fa5b5f8a78e986f7f
Oct 30 19:15:16.996803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:16.996765    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" event={"ID":"32c70801-a4d6-4502-8ec2-133dd5fb95da","Type":"ContainerStarted","Data":"fe99c0b3a7fe04239e5de01405a02f8e94345c9740d0076fa5b5f8a78e986f7f"}
Oct 30 19:15:18.005034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.004986    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" containerName="manager" containerID="cri-o://3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080" gracePeriod=2
Oct 30 19:15:18.299479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.299451    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:18.413856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.413830    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\" (UniqueName: \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\") pod \"e66b10f0-dafd-44fe-84bb-d38e98d47599\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") "
Oct 30 19:15:18.414053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.414034    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\") pod \"e66b10f0-dafd-44fe-84bb-d38e98d47599\" (UID: \"e66b10f0-dafd-44fe-84bb-d38e98d47599\") "
Oct 30 19:15:18.414267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.414245    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume" pod "e66b10f0-dafd-44fe-84bb-d38e98d47599" (UID: "e66b10f0-dafd-44fe-84bb-d38e98d47599"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:18.414399 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.414385    4320 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e66b10f0-dafd-44fe-84bb-d38e98d47599-extensions-socket-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:15:18.415979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.415961    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc" pod "e66b10f0-dafd-44fe-84bb-d38e98d47599" (UID: "e66b10f0-dafd-44fe-84bb-d38e98d47599"). InnerVolumeSpecName "kube-api-access-zwnmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.515860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:18.515782    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnmc\" (UniqueName: \"kubernetes.io/projected/e66b10f0-dafd-44fe-84bb-d38e98d47599-kube-api-access-zwnmc\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:15:19.011250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.011211    4320 generic.go:356] "Generic (PLEG): container finished" podID="e66b10f0-dafd-44fe-84bb-d38e98d47599" containerID="3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080" exitCode=2
Oct 30 19:15:19.011733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.011263    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:19.011733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.011319    4320 scope.go:122] "RemoveContainer" containerID="3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080"
Oct 30 19:15:19.013353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.013322    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" event={"ID":"32c70801-a4d6-4502-8ec2-133dd5fb95da","Type":"ContainerStarted","Data":"4f3ffc550e71a424fa9b4701e878839af04e67622c263f2fd903710d965c8b0f"}
Oct 30 19:15:19.013509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.013413    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:19.020067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.020050    4320 scope.go:122] "RemoveContainer" containerID="3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080"
Oct 30 19:15:19.020350 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:15:19.020332    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080\": container with ID starting with 3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080 not found: ID does not exist" containerID="3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080"
Oct 30 19:15:19.020396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.020360    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080"} err="failed to get container status \"3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080\": rpc error: code = NotFound desc = could not find container \"3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080\": container with ID starting with 3484e05f16dd52650ef042a73c6f40ba44d49c1d91c9c14aeb4ce48117dc3080 not found: ID does not exist"
Oct 30 19:15:19.025104 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:15:19.025079    4320 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" is forbidden: User \"system:node:ip-10-0-130-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-217.ec2.internal' and this object" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:19.095037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:19.094986    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" podStartSLOduration=1.744144022 podStartE2EDuration="4.094972333s" totalImagesPullingTime="2.350828311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.093618219 +0000 UTC m=+667.129130680" watchObservedRunningTime="2026-10-30 19:15:19.094972333 +0000 UTC m=+667.130484799"
Oct 30 19:15:19.107187 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:15:19.107150    4320 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-88lh5\" is forbidden: User \"system:node:ip-10-0-130-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-217.ec2.internal' and this object" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-88lh5"
Oct 30 19:15:20.094271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:20.094235    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" path="/var/lib/kubelet/pods/e66b10f0-dafd-44fe-84bb-d38e98d47599/volumes"
Oct 30 19:15:28.644668 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:15:28.644558    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:15:30.021101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:30.021067    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:42.949578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:42.949544    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:42.950162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:42.949792    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" podUID="32c70801-a4d6-4502-8ec2-133dd5fb95da" containerName="manager" containerID="cri-o://4f3ffc550e71a424fa9b4701e878839af04e67622c263f2fd903710d965c8b0f" gracePeriod=10
Oct 30 19:15:43.124713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.124676    4320 generic.go:356] "Generic (PLEG): container finished" podID="32c70801-a4d6-4502-8ec2-133dd5fb95da" containerID="4f3ffc550e71a424fa9b4701e878839af04e67622c263f2fd903710d965c8b0f" exitCode=0
Oct 30 19:15:43.124849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.124744    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" event={"ID":"32c70801-a4d6-4502-8ec2-133dd5fb95da","Type":"ContainerDied","Data":"4f3ffc550e71a424fa9b4701e878839af04e67622c263f2fd903710d965c8b0f"}
Oct 30 19:15:43.210361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.210306    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:43.238392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.238355    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\" (UniqueName: \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\") pod \"32c70801-a4d6-4502-8ec2-133dd5fb95da\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") "
Oct 30 19:15:43.238516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.238417    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\") pod \"32c70801-a4d6-4502-8ec2-133dd5fb95da\" (UID: \"32c70801-a4d6-4502-8ec2-133dd5fb95da\") "
Oct 30 19:15:43.238918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.238894    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume" pod "32c70801-a4d6-4502-8ec2-133dd5fb95da" (UID: "32c70801-a4d6-4502-8ec2-133dd5fb95da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.240485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.240464    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv" pod "32c70801-a4d6-4502-8ec2-133dd5fb95da" (UID: "32c70801-a4d6-4502-8ec2-133dd5fb95da"). InnerVolumeSpecName "kube-api-access-5nlfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.339502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.339466    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlfv\" (UniqueName: \"kubernetes.io/projected/32c70801-a4d6-4502-8ec2-133dd5fb95da-kube-api-access-5nlfv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:15:43.339502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:43.339495    4320 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32c70801-a4d6-4502-8ec2-133dd5fb95da-extensions-socket-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:15:44.130691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:44.130657    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m" event={"ID":"32c70801-a4d6-4502-8ec2-133dd5fb95da","Type":"ContainerDied","Data":"fe99c0b3a7fe04239e5de01405a02f8e94345c9740d0076fa5b5f8a78e986f7f"}
Oct 30 19:15:44.130691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:44.130676    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"
Oct 30 19:15:44.131185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:44.130718    4320 scope.go:122] "RemoveContainer" containerID="4f3ffc550e71a424fa9b4701e878839af04e67622c263f2fd903710d965c8b0f"
Oct 30 19:15:44.195057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:44.195021    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:44.209671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:44.209647    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-lbw9m"]
Oct 30 19:15:46.095106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:46.095068    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c70801-a4d6-4502-8ec2-133dd5fb95da" path="/var/lib/kubelet/pods/32c70801-a4d6-4502-8ec2-133dd5fb95da/volumes"
Oct 30 19:15:47.248319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248287    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248697    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" containerName="manager"
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248710    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" containerName="manager"
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248735    4320 cpu_manager.go:414] "containerMap: removing container" podUID="32c70801-a4d6-4502-8ec2-133dd5fb95da" containerName="manager"
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248740    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c70801-a4d6-4502-8ec2-133dd5fb95da" containerName="manager"
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248801    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66b10f0-dafd-44fe-84bb-d38e98d47599" containerName="manager"
Oct 30 19:15:47.248850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.248811    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c70801-a4d6-4502-8ec2-133dd5fb95da" containerName="manager"
Oct 30 19:15:47.253334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.253311    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.267584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.267562    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bw5jw\""
Oct 30 19:15:47.281676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.281651    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:15:47.376836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.376797    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.376836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.376836    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.376968    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.376986    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.377019    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.377049    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.377122    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2bt\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.377142    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.377164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.377156    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478327    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2bt\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478371    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478398    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478449    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478470    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478558    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.478667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478585    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478702    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478751    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.478908    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.479007    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.479069    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.479370    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.479509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.479409    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.481063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.481037    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.481173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.481150    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.497863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.497841    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.497863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.497855    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2bt\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\") pod \"maas-default-gateway-openshift-default-6f54779779-cnvpn\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.567950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.567885    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:47.881022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.880943    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:15:47.884481 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:15:47.884449    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15545d8b_27bf_4756_86e0_ce9a7d4b82d1.slice/crio-8b271fc223f84e5f18c3ee58a9dcb00595fdf1779ef847faee23aa80d7888aa4 WatchSource:0}: Error finding container 8b271fc223f84e5f18c3ee58a9dcb00595fdf1779ef847faee23aa80d7888aa4: Status 404 returned error can't find the container with id 8b271fc223f84e5f18c3ee58a9dcb00595fdf1779ef847faee23aa80d7888aa4
Oct 30 19:15:47.886700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.886665    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:47.886777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.886736    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:47.886777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:47.886768    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:48.152453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:48.152416    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerStarted","Data":"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"}
Oct 30 19:15:48.152645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:48.152481    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerStarted","Data":"8b271fc223f84e5f18c3ee58a9dcb00595fdf1779ef847faee23aa80d7888aa4"}
Oct 30 19:15:48.196213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:48.196154    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podStartSLOduration=1.196136396 podStartE2EDuration="1.196136396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.195322525 +0000 UTC m=+696.230834989" watchObservedRunningTime="2026-10-30 19:15:48.196136396 +0000 UTC m=+696.231648921"
Oct 30 19:15:48.568552 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:48.568456    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:48.573360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:48.573335    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:49.157669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:49.157635    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:49.159072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:15:49.159052    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:15:58.644536 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:15:58.644492    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:16:03.587876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.587834    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:03.591771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.591749    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:16:03.591943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.591914    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:03.595370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.595351    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.607867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.607845    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:03.609189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.609166    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:16:03.617610 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.617580    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:16:03.617752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.617676    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:16:03.617752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.617730    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fg2pp\""
Oct 30 19:16:03.617846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.617771    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:16:03.618787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.618765    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpcvt\""
Oct 30 19:16:03.682161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.682134    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:16:03.734991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.734966    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6faa0024-5182-47d7-96d2-043069294f99-config-file\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.735144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.735041    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4hp\" (UniqueName: \"kubernetes.io/projected/6faa0024-5182-47d7-96d2-043069294f99-kube-api-access-lt4hp\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.735144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.735128    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm959\" (UniqueName: \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\") pod \"authorino-56c6c9ddd5-qjmmr\" (UID: \"487751f8-79f6-4281-9a9f-9689e039f862\") " pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:03.836549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.836514    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4hp\" (UniqueName: \"kubernetes.io/projected/6faa0024-5182-47d7-96d2-043069294f99-kube-api-access-lt4hp\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.836549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.836562    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm959\" (UniqueName: \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\") pod \"authorino-56c6c9ddd5-qjmmr\" (UID: \"487751f8-79f6-4281-9a9f-9689e039f862\") " pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:03.836798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.836607    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6faa0024-5182-47d7-96d2-043069294f99-config-file\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.837219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.837199    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6faa0024-5182-47d7-96d2-043069294f99-config-file\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.856188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.856121    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4hp\" (UniqueName: \"kubernetes.io/projected/6faa0024-5182-47d7-96d2-043069294f99-kube-api-access-lt4hp\") pod \"limitador-limitador-7c9b87d574-mvtfx\" (UID: \"6faa0024-5182-47d7-96d2-043069294f99\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:03.856188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.856146    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm959\" (UniqueName: \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\") pod \"authorino-56c6c9ddd5-qjmmr\" (UID: \"487751f8-79f6-4281-9a9f-9689e039f862\") " pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:03.922092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.922061    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:03.928371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:03.928349    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:04.218371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:04.218179    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:04.221130 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:16:04.221099    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod487751f8_79f6_4281_9a9f_9689e039f862.slice/crio-32659c27e70d7ae9efd9b1c31bd4461f29ec247e5b92ba9c15c2d533dd0bb1fd WatchSource:0}: Error finding container 32659c27e70d7ae9efd9b1c31bd4461f29ec247e5b92ba9c15c2d533dd0bb1fd: Status 404 returned error can't find the container with id 32659c27e70d7ae9efd9b1c31bd4461f29ec247e5b92ba9c15c2d533dd0bb1fd
Oct 30 19:16:04.227342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:04.227319    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" event={"ID":"487751f8-79f6-4281-9a9f-9689e039f862","Type":"ContainerStarted","Data":"32659c27e70d7ae9efd9b1c31bd4461f29ec247e5b92ba9c15c2d533dd0bb1fd"}
Oct 30 19:16:04.232089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:04.232067    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:16:04.234874 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:16:04.234851    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faa0024_5182_47d7_96d2_043069294f99.slice/crio-028810fc6044183d9b9ba214610de43b387a1e208fe40d30581ec9bd13fec5fa WatchSource:0}: Error finding container 028810fc6044183d9b9ba214610de43b387a1e208fe40d30581ec9bd13fec5fa: Status 404 returned error can't find the container with id 028810fc6044183d9b9ba214610de43b387a1e208fe40d30581ec9bd13fec5fa
Oct 30 19:16:05.233190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:05.233149    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx" event={"ID":"6faa0024-5182-47d7-96d2-043069294f99","Type":"ContainerStarted","Data":"445fe27b7577b72ed0d922f7bbc57faaed0a109b326108cb6c9a0f4cbf25a1e0"}
Oct 30 19:16:05.233590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:05.233216    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx" event={"ID":"6faa0024-5182-47d7-96d2-043069294f99","Type":"ContainerStarted","Data":"028810fc6044183d9b9ba214610de43b387a1e208fe40d30581ec9bd13fec5fa"}
Oct 30 19:16:05.233590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:05.233273    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:05.271763 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:05.271713    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx" podStartSLOduration=1.399683543 podStartE2EDuration="2.271693361s" totalImagesPullingTime="872.009818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:05.2703315 +0000 UTC m=+713.305843963" watchObservedRunningTime="2026-10-30 19:16:05.271693361 +0000 UTC m=+713.307205882"
Oct 30 19:16:06.239007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:06.238913    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" event={"ID":"487751f8-79f6-4281-9a9f-9689e039f862","Type":"ContainerStarted","Data":"a3c4c7574e3354fe16e8cf128ff5f1efe060f4de68cf1f0fef324cd24219c389"}
Oct 30 19:16:06.276002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:06.275953    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" podStartSLOduration=1.8180215309999999 podStartE2EDuration="3.275940051s" totalImagesPullingTime="1.45791852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:06.275208332 +0000 UTC m=+714.310720796" watchObservedRunningTime="2026-10-30 19:16:06.275940051 +0000 UTC m=+714.311452515"
Oct 30 19:16:16.239659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:16.239627    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"
Oct 30 19:16:28.643787 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:16:28.643741    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:16:37.214534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.214489    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:37.215090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.214738    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" podUID="487751f8-79f6-4281-9a9f-9689e039f862" containerName="authorino" containerID="cri-o://a3c4c7574e3354fe16e8cf128ff5f1efe060f4de68cf1f0fef324cd24219c389" gracePeriod=30
Oct 30 19:16:37.383519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.383480    4320 generic.go:356] "Generic (PLEG): container finished" podID="487751f8-79f6-4281-9a9f-9689e039f862" containerID="a3c4c7574e3354fe16e8cf128ff5f1efe060f4de68cf1f0fef324cd24219c389" exitCode=0
Oct 30 19:16:37.383705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.383551    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" event={"ID":"487751f8-79f6-4281-9a9f-9689e039f862","Type":"ContainerDied","Data":"a3c4c7574e3354fe16e8cf128ff5f1efe060f4de68cf1f0fef324cd24219c389"}
Oct 30 19:16:37.474286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.474225    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:37.561989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.561963    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\" (UniqueName: \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\") pod \"487751f8-79f6-4281-9a9f-9689e039f862\" (UID: \"487751f8-79f6-4281-9a9f-9689e039f862\") "
Oct 30 19:16:37.563994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.563964    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959" pod "487751f8-79f6-4281-9a9f-9689e039f862" (UID: "487751f8-79f6-4281-9a9f-9689e039f862"). InnerVolumeSpecName "kube-api-access-nm959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:37.662985 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:37.662954    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm959\" (UniqueName: \"kubernetes.io/projected/487751f8-79f6-4281-9a9f-9689e039f862-kube-api-access-nm959\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:16:38.388833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:38.388799    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr" event={"ID":"487751f8-79f6-4281-9a9f-9689e039f862","Type":"ContainerDied","Data":"32659c27e70d7ae9efd9b1c31bd4461f29ec247e5b92ba9c15c2d533dd0bb1fd"}
Oct 30 19:16:38.389338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:38.388841    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qjmmr"
Oct 30 19:16:38.389338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:38.388858    4320 scope.go:122] "RemoveContainer" containerID="a3c4c7574e3354fe16e8cf128ff5f1efe060f4de68cf1f0fef324cd24219c389"
Oct 30 19:16:38.437149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:38.437115    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:38.438869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:38.438845    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qjmmr"]
Oct 30 19:16:40.094223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:16:40.094174    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487751f8-79f6-4281-9a9f-9689e039f862" path="/var/lib/kubelet/pods/487751f8-79f6-4281-9a9f-9689e039f862/volumes"
Oct 30 19:16:58.643893 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:16:58.643801    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:17:28.643630 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:17:28.643580    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:17:33.844112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:33.844049    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:17:42.678031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.677987    4320 generic.go:356] "Generic (PLEG): container finished" podID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerID="2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e" exitCode=0
Oct 30 19:17:42.678574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.678062    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerDied","Data":"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"}
Oct 30 19:17:42.678574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.678442    4320 scope.go:122] "RemoveContainer" containerID="2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"
Oct 30 19:17:42.679651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.679584    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:17:42.679771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.679700    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:17:42.679771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:42.679734    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:17:43.701701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:43.701661    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerStarted","Data":"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"}
Oct 30 19:17:44.568659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:44.568585    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:17:44.585623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:44.575711    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:17:44.585623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:44.575787    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:17:45.589623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:45.584734    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:17:45.589623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:45.584807    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:17:46.586622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:46.582015    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:17:46.586622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:46.582088    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:17:47.583632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:47.579723    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:17:47.583632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:47.579814    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:17:48.569372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:48.569328    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:17:48.569586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:48.569406    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:17:49.158066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:49.158033    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:17:49.572394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:49.572315    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:17:49.730453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:17:49.730426    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:17:58.644570 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:17:58.644529    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:18:01.770324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.770288    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c757k"]
Oct 30 19:18:01.770749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.770734    4320 cpu_manager.go:414] "containerMap: removing container" podUID="487751f8-79f6-4281-9a9f-9689e039f862" containerName="authorino"
Oct 30 19:18:01.770792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.770751    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487751f8-79f6-4281-9a9f-9689e039f862" containerName="authorino"
Oct 30 19:18:01.770826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.770815    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="487751f8-79f6-4281-9a9f-9689e039f862" containerName="authorino"
Oct 30 19:18:01.774009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.773986    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:01.796814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.796781    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c757k"]
Oct 30 19:18:01.880854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.880804    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jwn\" (UniqueName: \"kubernetes.io/projected/6ab50bb9-d775-4460-bcff-e18d445dc2a3-kube-api-access-l8jwn\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:01.881035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.881009    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ab50bb9-d775-4460-bcff-e18d445dc2a3-cert\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:01.981727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.981688    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ab50bb9-d775-4460-bcff-e18d445dc2a3-cert\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:01.981900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.981746    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jwn\" (UniqueName: \"kubernetes.io/projected/6ab50bb9-d775-4460-bcff-e18d445dc2a3-kube-api-access-l8jwn\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:01.984239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:01.984214    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ab50bb9-d775-4460-bcff-e18d445dc2a3-cert\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:02.000842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:02.000812    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jwn\" (UniqueName: \"kubernetes.io/projected/6ab50bb9-d775-4460-bcff-e18d445dc2a3-kube-api-access-l8jwn\") pod \"llmisvc-controller-manager-6466c554d8-c757k\" (UID: \"6ab50bb9-d775-4460-bcff-e18d445dc2a3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:02.087093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:02.087001    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:02.386526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:02.386502    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c757k"]
Oct 30 19:18:02.389237 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:18:02.389206    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab50bb9_d775_4460_bcff_e18d445dc2a3.slice/crio-086b4a7eb384f7002f896a083c95d8739c5b46c954ef9fbb19647dbb6953597e WatchSource:0}: Error finding container 086b4a7eb384f7002f896a083c95d8739c5b46c954ef9fbb19647dbb6953597e: Status 404 returned error can't find the container with id 086b4a7eb384f7002f896a083c95d8739c5b46c954ef9fbb19647dbb6953597e
Oct 30 19:18:02.390490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:02.390466    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:18:02.784765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:02.784734    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k" event={"ID":"6ab50bb9-d775-4460-bcff-e18d445dc2a3","Type":"ContainerStarted","Data":"086b4a7eb384f7002f896a083c95d8739c5b46c954ef9fbb19647dbb6953597e"}
Oct 30 19:18:03.791134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:03.791077    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k" event={"ID":"6ab50bb9-d775-4460-bcff-e18d445dc2a3","Type":"ContainerStarted","Data":"f01350edcee1811981d313697418155b3ca5183e600850d82933b9da21528e88"}
Oct 30 19:18:03.791643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:03.791318    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:03.830354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:03.830305    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k" podStartSLOduration=2.480339233 podStartE2EDuration="2.83029097s" totalImagesPullingTime="349.951737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:03.827984841 +0000 UTC m=+831.863497304" watchObservedRunningTime="2026-10-30 19:18:03.83029097 +0000 UTC m=+831.865803434"
Oct 30 19:18:28.644856 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:18:28.644813    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:18:34.796744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:34.796716    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c757k"
Oct 30 19:18:34.859745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:34.859704    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:18:34.859945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:34.859921    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" podUID="5b785e21-5e53-4147-8433-ab9381dfa0c1" containerName="manager" containerID="cri-o://2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05" gracePeriod=30
Oct 30 19:18:35.122772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.122745    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:18:35.192407 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.192379    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\" (UniqueName: \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\") pod \"5b785e21-5e53-4147-8433-ab9381dfa0c1\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") "
Oct 30 19:18:35.192564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.192534    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\" (UniqueName: \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\") pod \"5b785e21-5e53-4147-8433-ab9381dfa0c1\" (UID: \"5b785e21-5e53-4147-8433-ab9381dfa0c1\") "
Oct 30 19:18:35.194672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.194640    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert" pod "5b785e21-5e53-4147-8433-ab9381dfa0c1" (UID: "5b785e21-5e53-4147-8433-ab9381dfa0c1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:35.194796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.194673    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n" pod "5b785e21-5e53-4147-8433-ab9381dfa0c1" (UID: "5b785e21-5e53-4147-8433-ab9381dfa0c1"). InnerVolumeSpecName "kube-api-access-pwj4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:35.294140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.294109    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwj4n\" (UniqueName: \"kubernetes.io/projected/5b785e21-5e53-4147-8433-ab9381dfa0c1-kube-api-access-pwj4n\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:18:35.294140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.294134    4320 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b785e21-5e53-4147-8433-ab9381dfa0c1-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:18:35.933451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.933414    4320 generic.go:356] "Generic (PLEG): container finished" podID="5b785e21-5e53-4147-8433-ab9381dfa0c1" containerID="2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05" exitCode=0
Oct 30 19:18:35.933911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.933462    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"
Oct 30 19:18:35.933911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.933497    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" event={"ID":"5b785e21-5e53-4147-8433-ab9381dfa0c1","Type":"ContainerDied","Data":"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"}
Oct 30 19:18:35.933911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.933552    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b" event={"ID":"5b785e21-5e53-4147-8433-ab9381dfa0c1","Type":"ContainerDied","Data":"1e658fcd5690fb2a934db4c7f34feed7b14f234955a40d869b7dd11f416de8aa"}
Oct 30 19:18:35.933911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.933577    4320 scope.go:122] "RemoveContainer" containerID="2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"
Oct 30 19:18:35.942485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.942468    4320 scope.go:122] "RemoveContainer" containerID="2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"
Oct 30 19:18:35.942775 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:18:35.942755    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05\": container with ID starting with 2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05 not found: ID does not exist" containerID="2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"
Oct 30 19:18:35.942854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.942785    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05"} err="failed to get container status \"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05\": rpc error: code = NotFound desc = could not find container \"2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05\": container with ID starting with 2612a23c392d277263461897fe456712bdd40037590d0b95b990cdcecbed1f05 not found: ID does not exist"
Oct 30 19:18:35.995358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:35.995331    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:18:36.007269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:36.007246    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sjv8b"]
Oct 30 19:18:36.094078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:36.094050    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b785e21-5e53-4147-8433-ab9381dfa0c1" path="/var/lib/kubelet/pods/5b785e21-5e53-4147-8433-ab9381dfa0c1/volumes"
Oct 30 19:18:54.831698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:54.831658    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:18:57.558949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:18:57.558911    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:18:58.644011 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:18:58.643967    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:19:02.412811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:02.412775    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:19:08.922346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:08.922310    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:19:13.012349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:13.012314    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:19:28.643428 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:19:28.643388    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:19:32.973033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.972998    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"]
Oct 30 19:19:32.973451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.973386    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5b785e21-5e53-4147-8433-ab9381dfa0c1" containerName="manager"
Oct 30 19:19:32.973451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.973395    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b785e21-5e53-4147-8433-ab9381dfa0c1" containerName="manager"
Oct 30 19:19:32.973538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.973455    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b785e21-5e53-4147-8433-ab9381dfa0c1" containerName="manager"
Oct 30 19:19:32.976473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.976457    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:32.998968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:32.998940    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"]
Oct 30 19:19:33.107464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107437    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwjc\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-kube-api-access-gnwjc\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107469    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cede89a0-f33e-4db3-bcac-24ce856b418f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107489    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107505    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107581    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107676    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107737    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107796    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.107876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.107827    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208505    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwjc\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-kube-api-access-gnwjc\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208543    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cede89a0-f33e-4db3-bcac-24ce856b418f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208567    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208617    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208643    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208691    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.208779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208753    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208897    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.208951    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.209195    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.209230    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.209267    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.209302    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cede89a0-f33e-4db3-bcac-24ce856b418f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.209365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.209323    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.211334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.211313    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.211591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.211572    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.227533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.227482    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.227694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.227679    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwjc\" (UniqueName: \"kubernetes.io/projected/cede89a0-f33e-4db3-bcac-24ce856b418f-kube-api-access-gnwjc\") pod \"maas-default-gateway-openshift-default-6f54779779-j8c67\" (UID: \"cede89a0-f33e-4db3-bcac-24ce856b418f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.291631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.291593    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:33.607934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.607854    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"]
Oct 30 19:19:33.610900 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:19:33.610871    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcede89a0_f33e_4db3_bcac_24ce856b418f.slice/crio-78322318cc3aa06877d3d893e23e7cafc4b7ff02bc343be0a81c708fa514f28d WatchSource:0}: Error finding container 78322318cc3aa06877d3d893e23e7cafc4b7ff02bc343be0a81c708fa514f28d: Status 404 returned error can't find the container with id 78322318cc3aa06877d3d893e23e7cafc4b7ff02bc343be0a81c708fa514f28d
Oct 30 19:19:33.613048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.613008    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:33.613127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.613082    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:33.613127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:33.613111    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:34.210076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.210035    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" event={"ID":"cede89a0-f33e-4db3-bcac-24ce856b418f","Type":"ContainerStarted","Data":"f5934dc08b5a531686e82ec6355727cc084a13e42d7238d339a0335853f5c95b"}
Oct 30 19:19:34.210524 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.210101    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" event={"ID":"cede89a0-f33e-4db3-bcac-24ce856b418f","Type":"ContainerStarted","Data":"78322318cc3aa06877d3d893e23e7cafc4b7ff02bc343be0a81c708fa514f28d"}
Oct 30 19:19:34.287621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.279068    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podStartSLOduration=2.279049528 podStartE2EDuration="2.279049528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:34.262134931 +0000 UTC m=+922.297647394" watchObservedRunningTime="2026-10-30 19:19:34.279049528 +0000 UTC m=+922.314561995"
Oct 30 19:19:34.291785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.291754    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:34.295796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.295755    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:34.295957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:34.295824    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:35.292035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:35.291996    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:35.292635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:35.292584    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:36.294115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:36.294075    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.294519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:36.294143    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:37.294129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:37.294084    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:37.294555 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:37.294177    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:38.292624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:38.292087    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:38.292624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:38.292170    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:39.293750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:39.293703    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:39.294185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:39.293765    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:40.292927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:40.292889    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:19:40.293113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:40.292960    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:19:41.997343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:41.997308    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:41.997864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:41.997771    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:41.998793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:19:41.998766    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67"
Oct 30 19:19:58.644583 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:19:58.644499    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:20:28.643488 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:20:28.643453    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:20:58.645473 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:20:58.645435    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:21:26.941663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:26.941618    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:21:28.567253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:28.567197    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" podUID="32feda196b1df490b5d951791a17beab" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 19:21:28.644137 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:21:28.644098    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:21:30.817334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.817293    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 19:21:30.821139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.821113    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.837930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.837902    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 19:21:30.847950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.847926    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e67b84\""
Oct 30 19:21:30.865541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.865501    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgrt\" (UniqueName: \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.865766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.865742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.967364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.967330    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgrt\" (UniqueName: \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.967556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.967379    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.968080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.968058    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:30.986677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:30.986652    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgrt\" (UniqueName: \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\") pod \"payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:31.133752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:31.133726    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:31.430900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:31.430874    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 19:21:31.433852 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:31.433817    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0784043_9b3e_45ff_b204_de0a2d3555e0.slice/crio-1dc64935e744bf674f1520280cf92fae080c6d39d624d9c8fc7ab218bb58760c WatchSource:0}: Error finding container 1dc64935e744bf674f1520280cf92fae080c6d39d624d9c8fc7ab218bb58760c: Status 404 returned error can't find the container with id 1dc64935e744bf674f1520280cf92fae080c6d39d624d9c8fc7ab218bb58760c
Oct 30 19:21:31.752958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:31.752865    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" event={"ID":"c0784043-9b3e-45ff-b204-de0a2d3555e0","Type":"ContainerStarted","Data":"1dc64935e744bf674f1520280cf92fae080c6d39d624d9c8fc7ab218bb58760c"}
Oct 30 19:21:33.549542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.549452    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 19:21:33.553122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.553105    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.578773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.578735    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b97379\""
Oct 30 19:21:33.580650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.580623    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 19:21:33.581670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.581648    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b97379-dockercfg-2r4cs\""
Oct 30 19:21:33.696385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.696349    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.696569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.696404    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgw8\" (UniqueName: \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.764835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.764794    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" event={"ID":"c0784043-9b3e-45ff-b204-de0a2d3555e0","Type":"ContainerStarted","Data":"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"}
Oct 30 19:21:33.765036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.764924    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:33.797180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.797143    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgw8\" (UniqueName: \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.797372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.797272    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.798000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.797979    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.807285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.807198    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" podStartSLOduration=1.94258057 podStartE2EDuration="3.807185204s" totalImagesPullingTime="1.864604634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:33.805363996 +0000 UTC m=+1041.840876470" watchObservedRunningTime="2026-10-30 19:21:33.807185204 +0000 UTC m=+1041.842697668"
Oct 30 19:21:33.825357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.825329    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgw8\" (UniqueName: \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\") pod \"payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:33.866268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:33.866239    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:34.171061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.171025    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 19:21:34.174303 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:34.174271    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda82e62d_2257_4445_a27c_b95bb9d14e10.slice/crio-0b303fa43a8acba2975a484fa889f75628b747ff64bb642372ce3ee2acaf6774 WatchSource:0}: Error finding container 0b303fa43a8acba2975a484fa889f75628b747ff64bb642372ce3ee2acaf6774: Status 404 returned error can't find the container with id 0b303fa43a8acba2975a484fa889f75628b747ff64bb642372ce3ee2acaf6774
Oct 30 19:21:34.564772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.564683    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:21:34.568447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.568423    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.585932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.585888    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-249dd4\""
Oct 30 19:21:34.586093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.586079    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-249dd4-dockercfg-4jj25\""
Oct 30 19:21:34.600229 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.600191    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:21:34.644040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.643993    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:21:34.649538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.649511    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.669643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.667832    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:21:34.708570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.708534    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlrl\" (UniqueName: \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.708795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.708583    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.708795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.708761    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.708921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.708809    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzs48\" (UniqueName: \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.771102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.771056    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" event={"ID":"da82e62d-2257-4445-a27c-b95bb9d14e10","Type":"ContainerStarted","Data":"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"}
Oct 30 19:21:34.771441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.771115    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" event={"ID":"da82e62d-2257-4445-a27c-b95bb9d14e10","Type":"ContainerStarted","Data":"0b303fa43a8acba2975a484fa889f75628b747ff64bb642372ce3ee2acaf6774"}
Oct 30 19:21:34.771441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.771235    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:34.809665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.809628    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlrl\" (UniqueName: \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.809868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.809675    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.809868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.809728    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.809868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.809762    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzs48\" (UniqueName: \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.810576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.810551    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.810710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.810620    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.813383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.813341    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" podStartSLOduration=1.813326864 podStartE2EDuration="1.813326864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:34.811373489 +0000 UTC m=+1042.846885953" watchObservedRunningTime="2026-10-30 19:21:34.813326864 +0000 UTC m=+1042.848839329"
Oct 30 19:21:34.833640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.833325    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlrl\" (UniqueName: \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\") pod \"payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.833640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.833451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzs48\" (UniqueName: \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\") pod \"payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:34.887769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.887731    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:34.970534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:34.970159    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:35.186718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.186685    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:21:35.189946 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:35.189868    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861d5d2b_e377_4710_9f93_f2e79194440a.slice/crio-0064ea6b9be77feef2d1f345f6aab053ef867bea32d7847babc68a8f45cd4933 WatchSource:0}: Error finding container 0064ea6b9be77feef2d1f345f6aab053ef867bea32d7847babc68a8f45cd4933: Status 404 returned error can't find the container with id 0064ea6b9be77feef2d1f345f6aab053ef867bea32d7847babc68a8f45cd4933
Oct 30 19:21:35.274004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.273966    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:21:35.277574 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:35.277546    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40c2549_5f26_4370_b4fe_d214a17fb445.slice/crio-4fb65ac82c3eea6c15ef63fc2b7666f881eb9e5b4d4fef19c4aec6f928b71dac WatchSource:0}: Error finding container 4fb65ac82c3eea6c15ef63fc2b7666f881eb9e5b4d4fef19c4aec6f928b71dac: Status 404 returned error can't find the container with id 4fb65ac82c3eea6c15ef63fc2b7666f881eb9e5b4d4fef19c4aec6f928b71dac
Oct 30 19:21:35.566327 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.566239    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 19:21:35.569897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.569875    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.587324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.587302    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c5449c\""
Oct 30 19:21:35.590542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.589230    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 19:21:35.602249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.598281    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c5449c-dockercfg-g5nsp\""
Oct 30 19:21:35.721244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.721201    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.721436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.721413    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj92\" (UniqueName: \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.778216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.778174    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" event={"ID":"f40c2549-5f26-4370-b4fe-d214a17fb445","Type":"ContainerStarted","Data":"fc4d48547d520edd325c0c90e3e3c7275be01b43712858fc4b85c039685ba6de"}
Oct 30 19:21:35.778426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.778282    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:35.778426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.778308    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" event={"ID":"f40c2549-5f26-4370-b4fe-d214a17fb445","Type":"ContainerStarted","Data":"4fb65ac82c3eea6c15ef63fc2b7666f881eb9e5b4d4fef19c4aec6f928b71dac"}
Oct 30 19:21:35.779922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.779885    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" event={"ID":"861d5d2b-e377-4710-9f93-f2e79194440a","Type":"ContainerStarted","Data":"5ecfc22d398db1061b5f6c7697bb7ae69f904ba1831a5d23c8fb8233704db07f"}
Oct 30 19:21:35.780043 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.779949    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" event={"ID":"861d5d2b-e377-4710-9f93-f2e79194440a","Type":"ContainerStarted","Data":"0064ea6b9be77feef2d1f345f6aab053ef867bea32d7847babc68a8f45cd4933"}
Oct 30 19:21:35.780122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.780073    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:35.822586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.822497    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.822786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.822653    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj92\" (UniqueName: \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.823273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.823248    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.848690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.848637    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" podStartSLOduration=1.848622132 podStartE2EDuration="1.848622132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:35.845546597 +0000 UTC m=+1043.881059062" watchObservedRunningTime="2026-10-30 19:21:35.848622132 +0000 UTC m=+1043.884134610"
Oct 30 19:21:35.852218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.852189    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj92\" (UniqueName: \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\") pod \"payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.884215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.884185    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:35.909304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:35.909252    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" podStartSLOduration=1.9092372439999998 podStartE2EDuration="1.909237244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:35.906719071 +0000 UTC m=+1043.942231535" watchObservedRunningTime="2026-10-30 19:21:35.909237244 +0000 UTC m=+1043.944749701"
Oct 30 19:21:36.000430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.000387    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:21:36.183043 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.183010    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 19:21:36.186957 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:36.186925    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bc7621_6bef_42ca_a871_a96b5bac5269.slice/crio-b97bba6d5f60e1c679e5190741f7978f3d7328ec11b52700ba922eccaaecec05 WatchSource:0}: Error finding container b97bba6d5f60e1c679e5190741f7978f3d7328ec11b52700ba922eccaaecec05: Status 404 returned error can't find the container with id b97bba6d5f60e1c679e5190741f7978f3d7328ec11b52700ba922eccaaecec05
Oct 30 19:21:36.787178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.787138    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" event={"ID":"e2bc7621-6bef-42ca-a871-a96b5bac5269","Type":"ContainerStarted","Data":"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"}
Oct 30 19:21:36.787677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.787200    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" event={"ID":"e2bc7621-6bef-42ca-a871-a96b5bac5269","Type":"ContainerStarted","Data":"b97bba6d5f60e1c679e5190741f7978f3d7328ec11b52700ba922eccaaecec05"}
Oct 30 19:21:36.787677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.787374    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:36.825531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:36.825475    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" podStartSLOduration=1.825459591 podStartE2EDuration="1.825459591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:36.823805628 +0000 UTC m=+1044.859318093" watchObservedRunningTime="2026-10-30 19:21:36.825459591 +0000 UTC m=+1044.860972055"
Oct 30 19:21:37.756998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.756959    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 19:21:37.761048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.761026    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.792007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.791979    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"simulator-w2-9efc72-kserve-self-signed-certs\""
Oct 30 19:21:37.792424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.792021    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"openshift-service-ca.crt\""
Oct 30 19:21:37.792424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.792055    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"default-dockercfg-l9v2g\""
Oct 30 19:21:37.792424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.792164    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"kube-root-ca.crt\""
Oct 30 19:21:37.793956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.793936    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 19:21:37.840419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840383    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q95\" (UniqueName: \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.840639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840446    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.840639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840543    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.840770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840682    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.840832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840806    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.840886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.840851    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.941845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.941804    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.941994    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942036    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942094    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q95\" (UniqueName: \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942153    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942178    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942179    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942468    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.942550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.942524    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.944365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.944342    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.944508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.944491    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:37.960943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:37.960915    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q95\" (UniqueName: \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\") pod \"simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") " pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:38.091585 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:38.091487    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:38.396911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:38.396885    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 19:21:38.399096 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:38.399067    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd4fbb1b_5826_43f7_b333_2e2365ab305f.slice/crio-39e3c45d9e02f49cb17684e3b9ef179cb75a13f71ea6e04844918722a1e9beb2 WatchSource:0}: Error finding container 39e3c45d9e02f49cb17684e3b9ef179cb75a13f71ea6e04844918722a1e9beb2: Status 404 returned error can't find the container with id 39e3c45d9e02f49cb17684e3b9ef179cb75a13f71ea6e04844918722a1e9beb2
Oct 30 19:21:38.800030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:38.799906    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" event={"ID":"dd4fbb1b-5826-43f7-b333-2e2365ab305f","Type":"ContainerStarted","Data":"39e3c45d9e02f49cb17684e3b9ef179cb75a13f71ea6e04844918722a1e9beb2"}
Oct 30 19:21:39.806829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:39.806791    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" event={"ID":"dd4fbb1b-5826-43f7-b333-2e2365ab305f","Type":"ContainerStarted","Data":"4d88084238ba3d53fe9c3034154cd47aa7dfda902229e8f97c77fa5ec948399a"}
Oct 30 19:21:39.851806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:39.851695    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podStartSLOduration=1.6903832410000001 podStartE2EDuration="2.851678652s" totalImagesPullingTime="1.161295411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.849339309 +0000 UTC m=+1047.884851774" watchObservedRunningTime="2026-10-30 19:21:39.851678652 +0000 UTC m=+1047.887191116"
Oct 30 19:21:40.102581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:40.102501    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:41.830505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.830462    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 19:21:41.843746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.843720    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:41.848629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.848556    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 19:21:41.858392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.858363    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:21:41.862035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.862011    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c5449c-serving-cert\""
Oct 30 19:21:41.862035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.862030    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:21:41.862677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.862654    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:21:41.870163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.870138    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r84xv\""
Oct 30 19:21:41.998767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.998729    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:41.998986 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.998789    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjg8d\" (UniqueName: \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:41.999070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.999044    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:41.999163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:41.999146    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.100662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.100559    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.100662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.100629    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.100894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.100699    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.100894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.100738    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjg8d\" (UniqueName: \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.100894 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:21:42.100870    4320 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-serving-cert: secret "maas-api-e2e-worker-w0-c5449c-serving-cert" not found
Oct 30 19:21:42.101044 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:21:42.100955    4320 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls podName:7325a073-a722-4a52-92de-24cb960845db nodeName:}" failed. No retries permitted until 2026-10-30 19:21:42.600931087 +0000 UTC m=+1050.636443545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls") pod "maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" (UID: "7325a073-a722-4a52-92de-24cb960845db") : secret "maas-api-e2e-worker-w0-c5449c-serving-cert" not found
Oct 30 19:21:42.101346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.101324    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.103276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.103252    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.120705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.120684    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjg8d\" (UniqueName: \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.606547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.606512    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.609011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.608981    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:42.764370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:42.764337    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:43.065993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:43.065968    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 19:21:43.068053 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:43.068020    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7325a073_a722_4a52_92de_24cb960845db.slice/crio-f8e7f4c6eae01910af987b454ba2120b7d094f3ebb31c2bc3ec4b23cc231e453 WatchSource:0}: Error finding container f8e7f4c6eae01910af987b454ba2120b7d094f3ebb31c2bc3ec4b23cc231e453: Status 404 returned error can't find the container with id f8e7f4c6eae01910af987b454ba2120b7d094f3ebb31c2bc3ec4b23cc231e453
Oct 30 19:21:43.857010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:43.856926    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" event={"ID":"7325a073-a722-4a52-92de-24cb960845db","Type":"ContainerStarted","Data":"f8e7f4c6eae01910af987b454ba2120b7d094f3ebb31c2bc3ec4b23cc231e453"}
Oct 30 19:21:44.775290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:44.775259    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 19:21:45.783479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:45.783448    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 19:21:45.871382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:45.871349    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" event={"ID":"7325a073-a722-4a52-92de-24cb960845db","Type":"ContainerStarted","Data":"837790640aa43186118e290c921d7b76ce0b022f97ecf640f3aff7ef699d1b20"}
Oct 30 19:21:45.871589 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:45.871477    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:45.911360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:45.911300    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" podStartSLOduration=2.656233076 podStartE2EDuration="4.911282396s" totalImagesPullingTime="2.25504932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:45.909228819 +0000 UTC m=+1053.944741280" watchObservedRunningTime="2026-10-30 19:21:45.911282396 +0000 UTC m=+1053.946794893"
Oct 30 19:21:46.746705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:46.746668    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:21:46.793534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:46.793500    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:21:46.796833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:46.796805    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:21:47.796121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:47.796081    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 19:21:48.096014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:48.095919    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:48.106133 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:48.106104    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:48.906998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:48.906970    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:21:49.269224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.269144    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:21:49.273496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.273473    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.291436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.291076    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"default-dockercfg-bs5pk\""
Oct 30 19:21:49.291436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.291154    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"openshift-service-ca.crt\""
Oct 30 19:21:49.291436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.291269    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"kube-root-ca.crt\""
Oct 30 19:21:49.291436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.291406    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"simulator-w6-6e0c55-kserve-self-signed-certs\""
Oct 30 19:21:49.295181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.295155    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:21:49.382326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382282    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.382543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382338    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjznv\" (UniqueName: \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.382543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382476    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.382715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382553    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.382715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382589    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.382824 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.382774    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484061    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484102    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484199    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484242    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484284    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjznv\" (UniqueName: \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484321    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484692    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484750    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.484842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.484752    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.486517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.486497    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.486793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.486772    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.503003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.502974    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjznv\" (UniqueName: \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\") pod \"simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") " pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.589786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.589682    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:49.900325 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:49.900291    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:21:49.906647 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:49.906616    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82fd157c_26f2_489a_bc7c_8c48034534bb.slice/crio-68c92c0624cfbb0b9e27710e4505420a36d720b4237c793df00fb22c7e30556f WatchSource:0}: Error finding container 68c92c0624cfbb0b9e27710e4505420a36d720b4237c793df00fb22c7e30556f: Status 404 returned error can't find the container with id 68c92c0624cfbb0b9e27710e4505420a36d720b4237c793df00fb22c7e30556f
Oct 30 19:21:50.904941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:50.904898    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" event={"ID":"82fd157c-26f2-489a-bc7c-8c48034534bb","Type":"ContainerStarted","Data":"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"}
Oct 30 19:21:50.904941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:50.904945    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" event={"ID":"82fd157c-26f2-489a-bc7c-8c48034534bb","Type":"ContainerStarted","Data":"68c92c0624cfbb0b9e27710e4505420a36d720b4237c793df00fb22c7e30556f"}
Oct 30 19:21:50.945414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:50.945348    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podStartSLOduration=1.945326368 podStartE2EDuration="1.945326368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:50.942911113 +0000 UTC m=+1058.978423568" watchObservedRunningTime="2026-10-30 19:21:50.945326368 +0000 UTC m=+1058.980838833"
Oct 30 19:21:51.590037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:51.590000    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:51.884427 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:51.884395    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 19:21:53.770987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:53.770943    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:21:54.876676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.876638    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:21:54.881524 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.881499    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.895569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.895538    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c5449c-openshift-default-dockercfg-k6524\""
Oct 30 19:21:54.909867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.909840    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:21:54.946108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946076    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946132    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946271    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946313    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946387    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946436    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946472    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946515    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:54.946590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:54.946545    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzr8\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.047949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.047910    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.047958    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.047990    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048025    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048060    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048081    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048113    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzr8\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048186    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048229    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048383    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.048914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048882    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.049017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.048921    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.049017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.049003    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.049130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.049108    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.050476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.050451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.050836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.050814    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.067368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.067330    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzr8\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.067473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.067382    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.208245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.208210    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:55.526579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.526552    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:21:55.529085 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:55.529055    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4159106d_5f94_484b_b1dd_e843018750f6.slice/crio-6c028506aa53bd06d8f6d075ab457f7c540edf413fceeb1924f2413abee98925 WatchSource:0}: Error finding container 6c028506aa53bd06d8f6d075ab457f7c540edf413fceeb1924f2413abee98925: Status 404 returned error can't find the container with id 6c028506aa53bd06d8f6d075ab457f7c540edf413fceeb1924f2413abee98925
Oct 30 19:21:55.531560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.531523    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:21:55.531688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.531620    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:21:55.531688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.531666    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:21:55.663515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.663083    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 19:21:55.668493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.668468    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.683873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.683844    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"openshift-service-ca.crt\""
Oct 30 19:21:55.684815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.684788    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"kube-root-ca.crt\""
Oct 30 19:21:55.687879 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.687854    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 19:21:55.695528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.695507    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"default-dockercfg-2pfzk\""
Oct 30 19:21:55.695683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.695574    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"simulator-w1-5ac614-kserve-self-signed-certs\""
Oct 30 19:21:55.757506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757475    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.757708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757524    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.757708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757558    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.757791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757735    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.757863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757843    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.757942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.757927    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54vq\" (UniqueName: \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.858945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.858849    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54vq\" (UniqueName: \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.858945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.858919    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.858973    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859016    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859085    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859178    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859409    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859433    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.859547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.859522    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.861550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.861520    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.861720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.861678    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.877856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.877823    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54vq\" (UniqueName: \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\") pod \"simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") " pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:55.933590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.933555    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" event={"ID":"4159106d-5f94-484b-b1dd-e843018750f6","Type":"ContainerStarted","Data":"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"}
Oct 30 19:21:55.933590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.933621    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" event={"ID":"4159106d-5f94-484b-b1dd-e843018750f6","Type":"ContainerStarted","Data":"6c028506aa53bd06d8f6d075ab457f7c540edf413fceeb1924f2413abee98925"}
Oct 30 19:21:55.977633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.977552    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podStartSLOduration=1.977501368 podStartE2EDuration="1.977501368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:55.974703851 +0000 UTC m=+1064.010216316" watchObservedRunningTime="2026-10-30 19:21:55.977501368 +0000 UTC m=+1064.013013836"
Oct 30 19:21:55.986312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:55.986270    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:56.209375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.209332    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:21:56.211472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.211234    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.211472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.211294    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:21:56.319736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.319701    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 19:21:56.322955 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:21:56.322928    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54fab2ed_de59_4829_88f8_547e0c87e610.slice/crio-c764dd34c58d90cee565db73e98eb7071ae81ee2f92bbe38d5f6c284c9f37307 WatchSource:0}: Error finding container c764dd34c58d90cee565db73e98eb7071ae81ee2f92bbe38d5f6c284c9f37307: Status 404 returned error can't find the container with id c764dd34c58d90cee565db73e98eb7071ae81ee2f92bbe38d5f6c284c9f37307
Oct 30 19:21:56.939804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.939766    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" event={"ID":"54fab2ed-de59-4829-88f8-547e0c87e610","Type":"ContainerStarted","Data":"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"}
Oct 30 19:21:56.940212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.939822    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" event={"ID":"54fab2ed-de59-4829-88f8-547e0c87e610","Type":"ContainerStarted","Data":"c764dd34c58d90cee565db73e98eb7071ae81ee2f92bbe38d5f6c284c9f37307"}
Oct 30 19:21:56.986621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.982337    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podStartSLOduration=1.9823171 podStartE2EDuration="1.9823171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:56.982022711 +0000 UTC m=+1065.017535176" watchObservedRunningTime="2026-10-30 19:21:56.9823171 +0000 UTC m=+1065.017829564"
Oct 30 19:21:56.994838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:56.994791    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:21:57.209654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:57.209553    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.209654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:57.209630    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:21:58.209555 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:58.209517    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:21:58.209987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:58.209583    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:21:58.644806 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:21:58.644769    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:21:59.209469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:59.209437    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:21:59.209665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:59.209488    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:21:59.593876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:59.593787    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:59.606807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:59.606775    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:21:59.966213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:21:59.966184    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:22:00.209332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:00.209299    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:22:00.209495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:00.209360    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:22:01.208751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:01.208712    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 19:22:01.209182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:01.208775    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 19:22:02.212930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:02.212900    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:22:02.213311 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:02.213162    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:22:02.213968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:02.213950    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:22:05.011075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.011043    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:22:05.882103 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.882067    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 19:22:05.889708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.889688    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.904535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.904509    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 19:22:05.913436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.913412    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"premium-w2-9efc72-kserve-self-signed-certs\""
Oct 30 19:22:05.987491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.987455    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:22:05.992563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992534    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.992718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992584    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.992718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992707    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtc5j\" (UniqueName: \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.992847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.992847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992843    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:05.992952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:05.992866    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.001300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.001277    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:22:06.094130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094084    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094138    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094370    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094414    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094469    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtc5j\" (UniqueName: \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094488    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094564    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094646    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.094993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.094858    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.097049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.097023    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.097350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.097334    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.114018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.113996    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtc5j\" (UniqueName: \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\") pod \"premium-w2-9efc72-kserve-59f6db8988-sk5rj\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") " pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.201859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.201839    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:06.494486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.494462    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 19:22:06.497049 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:22:06.497009    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e6bd84_7550_4aa1_b304_567e34d5c462.slice/crio-b5547fa40afd838b061b1437f34539a75cd07ccd486ad78ed1dfdc52f33a1503 WatchSource:0}: Error finding container b5547fa40afd838b061b1437f34539a75cd07ccd486ad78ed1dfdc52f33a1503: Status 404 returned error can't find the container with id b5547fa40afd838b061b1437f34539a75cd07ccd486ad78ed1dfdc52f33a1503
Oct 30 19:22:06.992804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.992763    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" event={"ID":"d9e6bd84-7550-4aa1-b304-567e34d5c462","Type":"ContainerStarted","Data":"ead3ca977c59bb201e4e4e70000f51820060b8bb116b20191334386227578406"}
Oct 30 19:22:06.992804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.992817    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" event={"ID":"d9e6bd84-7550-4aa1-b304-567e34d5c462","Type":"ContainerStarted","Data":"b5547fa40afd838b061b1437f34539a75cd07ccd486ad78ed1dfdc52f33a1503"}
Oct 30 19:22:06.998479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:06.998459    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:22:07.034832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:07.034794    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podStartSLOduration=2.034782245 podStartE2EDuration="2.034782245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:07.032413362 +0000 UTC m=+1075.067925826" watchObservedRunningTime="2026-10-30 19:22:07.034782245 +0000 UTC m=+1075.070294709"
Oct 30 19:22:07.202350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:07.202311    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:14.255867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:14.255831    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:22:16.202379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:16.202348    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:16.214737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:16.214715    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:17.054059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:17.054031    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:22:23.749887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:23.749854    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:22:24.602582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.602549    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:22:24.609282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.609256    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.634130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.634074    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:22:24.640996 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.640969    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-249dd4\"/\"default-dockercfg-tt746\""
Oct 30 19:22:24.641717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.641587    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-249dd4\"/\"kube-root-ca.crt\""
Oct 30 19:22:24.641717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.641703    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-249dd4\"/\"simulator-w3-249dd4-kserve-self-signed-certs\""
Oct 30 19:22:24.641791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.641779    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-249dd4\"/\"openshift-service-ca.crt\""
Oct 30 19:22:24.674502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674476    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.674655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674519    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk57s\" (UniqueName: \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.674655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674562    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.674767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674747    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.674831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674817    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.674930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.674908    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.776657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776589    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776696    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776836    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776864    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk57s\" (UniqueName: \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776910    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.776969    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.777039    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.777236    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.777456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.777336    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.779199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.779173    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.779572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.779544    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.796278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.796255    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk57s\" (UniqueName: \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\") pod \"simulator-w3-249dd4-kserve-b47dbc8dc-brh2j\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") " pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:24.923453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:24.923421    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:25.228068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:25.228040    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:22:25.230198 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:22:25.230166    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d690fa9_c8ce_4d82_a5be_fc7456679906.slice/crio-d6b5203f579769fd62bcb69ec254b9de69a0f9356a04b9ac44104bb8c6b15095 WatchSource:0}: Error finding container d6b5203f579769fd62bcb69ec254b9de69a0f9356a04b9ac44104bb8c6b15095: Status 404 returned error can't find the container with id d6b5203f579769fd62bcb69ec254b9de69a0f9356a04b9ac44104bb8c6b15095
Oct 30 19:22:26.094979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:26.094945    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" event={"ID":"8d690fa9-c8ce-4d82-a5be-fc7456679906","Type":"ContainerStarted","Data":"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"}
Oct 30 19:22:26.095348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:26.094987    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" event={"ID":"8d690fa9-c8ce-4d82-a5be-fc7456679906","Type":"ContainerStarted","Data":"d6b5203f579769fd62bcb69ec254b9de69a0f9356a04b9ac44104bb8c6b15095"}
Oct 30 19:22:26.132750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:26.132699    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podStartSLOduration=2.132685761 podStartE2EDuration="2.132685761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:26.130768331 +0000 UTC m=+1094.166280802" watchObservedRunningTime="2026-10-30 19:22:26.132685761 +0000 UTC m=+1094.168198225"
Oct 30 19:22:26.923667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:26.923636    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:28.643264 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:22:28.643231    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:22:29.813251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.813217    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 19:22:29.816456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.816434    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.830660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.830641    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9efc72-openshift-default-dockercfg-jrkpb\""
Oct 30 19:22:29.845695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.845670    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 19:22:29.924951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.924926    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.924972    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925069    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925175    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925204    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925292    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925349    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925372    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9szg\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:29.925486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:29.925392    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.025999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.025966    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9szg\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026016    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026059    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026117    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026165    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026231    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026268    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026349    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.026483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026392    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.027007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026628    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.027007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026722    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.027007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026911    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.027007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026967    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.027007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.026991    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.029112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.029088    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.029259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.029243    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.046309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.046282    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.046494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.046474    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9szg\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.128742 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.128664    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:30.444925 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.444900    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 19:22:30.447104 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:22:30.447078    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96044409_2c69_4def_8e3d_7385c86185db.slice/crio-2c55c5c9fcb857f8fbc10dcd0a90f0ccaf4018b3efc8f6444f6ca112050a00a8 WatchSource:0}: Error finding container 2c55c5c9fcb857f8fbc10dcd0a90f0ccaf4018b3efc8f6444f6ca112050a00a8: Status 404 returned error can't find the container with id 2c55c5c9fcb857f8fbc10dcd0a90f0ccaf4018b3efc8f6444f6ca112050a00a8
Oct 30 19:22:30.449328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.449297    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:30.449390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.449361    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:30.449430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:30.449390    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:31.118891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.118827    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" event={"ID":"96044409-2c69-4def-8e3d-7385c86185db","Type":"ContainerStarted","Data":"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"}
Oct 30 19:22:31.119289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.118934    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" event={"ID":"96044409-2c69-4def-8e3d-7385c86185db","Type":"ContainerStarted","Data":"2c55c5c9fcb857f8fbc10dcd0a90f0ccaf4018b3efc8f6444f6ca112050a00a8"}
Oct 30 19:22:31.129241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.129219    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:31.130948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.130925    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:31.131067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.130974    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:31.182620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:31.178392    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podStartSLOduration=2.178374928 podStartE2EDuration="2.178374928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:31.173077727 +0000 UTC m=+1099.208590191" watchObservedRunningTime="2026-10-30 19:22:31.178374928 +0000 UTC m=+1099.213887387"
Oct 30 19:22:32.129182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:32.129099    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:32.129182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:32.129157    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:33.129738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:33.129704    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:33.130479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:33.129776    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:34.130441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:34.129787    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:34.130441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:34.129841    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:34.923752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:34.923722    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:34.939009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:34.938980    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:35.133498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:35.133458    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:35.134136 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:35.134103    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:35.156787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:35.156722    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:22:36.130027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:36.129991    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:22:36.130274 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:36.130250    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 19:22:36.444282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:36.444249    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:22:37.394340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:37.394311    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:37.395161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:37.395131    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:37.396080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:37.396057    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 19:22:40.015627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.015566    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:22:40.021225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.021198    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.044590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.044560    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:22:40.106990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.106960    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hkg\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.106993    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107032    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107134    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107208    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107274    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107316    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107459    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.107515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.107495    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208206    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hkg\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208239    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208265    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208282    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208316    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208340    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208390    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208485    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208520    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208724    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208801    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208863    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.208966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.208944    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.209221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.209203    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.210753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.210727    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.211333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.211316    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.227966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.227938    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hkg\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.228066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.228014    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.338901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.338831    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:40.673137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.673110    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:22:40.674839 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:22:40.674809    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8cfcb8_103d_4119_8eec_2a8dcc31f861.slice/crio-7e3212a3b335e62c7cba57c2cd2de2aee7782b7a1c27d6418d548c7e49cfbf32 WatchSource:0}: Error finding container 7e3212a3b335e62c7cba57c2cd2de2aee7782b7a1c27d6418d548c7e49cfbf32: Status 404 returned error can't find the container with id 7e3212a3b335e62c7cba57c2cd2de2aee7782b7a1c27d6418d548c7e49cfbf32
Oct 30 19:22:40.677413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.677378    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:40.677498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.677444    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:40.677498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:40.677480    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:22:41.196698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.196654    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" event={"ID":"9c8cfcb8-103d-4119-8eec-2a8dcc31f861","Type":"ContainerStarted","Data":"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"}
Oct 30 19:22:41.197216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.196719    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" event={"ID":"9c8cfcb8-103d-4119-8eec-2a8dcc31f861","Type":"ContainerStarted","Data":"7e3212a3b335e62c7cba57c2cd2de2aee7782b7a1c27d6418d548c7e49cfbf32"}
Oct 30 19:22:41.261526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.261482    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podStartSLOduration=2.261468707 podStartE2EDuration="2.261468707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:41.249312897 +0000 UTC m=+1109.284825364" watchObservedRunningTime="2026-10-30 19:22:41.261468707 +0000 UTC m=+1109.296981170"
Oct 30 19:22:41.338956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.338921    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:41.340686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.340662    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:41.340801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:41.340704    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:42.339171 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:42.339138    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:42.339547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:42.339192    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:43.339927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:43.339891    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:43.342168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:43.339949    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:44.339959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:44.339927    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:44.340344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:44.339983    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:45.340102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:45.340067    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:45.340511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:45.340125    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:46.339897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:46.339861    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:22:46.340075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:46.339926    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:22:47.344175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:47.344140    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:47.344584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:47.344508    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:47.345234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:47.345214    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:22:52.847276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.847243    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 19:22:52.851765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.851741    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.881485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.881453    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"openshift-service-ca.crt\""
Oct 30 19:22:52.881654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.881556    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"premium-w0-c5449c-kserve-self-signed-certs\""
Oct 30 19:22:52.881654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.881626    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"kube-root-ca.crt\""
Oct 30 19:22:52.881654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.881463    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5449c\"/\"default-dockercfg-dwctm\""
Oct 30 19:22:52.885120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.885094    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 19:22:52.964266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964232    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.964435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964327    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.964483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964447    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.964518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964490    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.964569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964531    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:52.964693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:52.964666    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mj4\" (UniqueName: \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066290    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mj4\" (UniqueName: \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066372    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066445    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066532    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066571    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.066686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.066642    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.067066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.067038    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.067483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.067453    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.068009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.067980    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.069804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.069770    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.075620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.070587    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.086627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.085688    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mj4\" (UniqueName: \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\") pod \"premium-w0-c5449c-kserve-777559f966-zcjwg\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") " pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.167464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.167435    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:53.278583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.278550    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:22:53.465900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:53.465799    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 19:22:53.471738 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:22:53.471713    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c11a744_2060_412d_9f50_62a2b28bb993.slice/crio-073d941339add4f67105a323f494c0853ea7e36cde0db290d8a73a4bd7b49a51 WatchSource:0}: Error finding container 073d941339add4f67105a323f494c0853ea7e36cde0db290d8a73a4bd7b49a51: Status 404 returned error can't find the container with id 073d941339add4f67105a323f494c0853ea7e36cde0db290d8a73a4bd7b49a51
Oct 30 19:22:54.287160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:54.287117    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" event={"ID":"6c11a744-2060-412d-9f50-62a2b28bb993","Type":"ContainerStarted","Data":"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"}
Oct 30 19:22:54.287728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:54.287171    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" event={"ID":"6c11a744-2060-412d-9f50-62a2b28bb993","Type":"ContainerStarted","Data":"073d941339add4f67105a323f494c0853ea7e36cde0db290d8a73a4bd7b49a51"}
Oct 30 19:22:54.327767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:54.327717    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podStartSLOduration=2.32770403 podStartE2EDuration="2.32770403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:54.32750626 +0000 UTC m=+1122.363018724" watchObservedRunningTime="2026-10-30 19:22:54.32770403 +0000 UTC m=+1122.363216500"
Oct 30 19:22:55.168301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:22:55.168229    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:22:58.644796 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:22:58.644759    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:23:03.168547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:03.168509    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:23:03.193439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:03.193392    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:23:03.359651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:03.359522    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:23:08.291886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.291854    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"]
Oct 30 19:23:08.295752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.295736    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.314062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.314038    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"]
Oct 30 19:23:08.322834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.322809    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:08.322971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.322901    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:23:08.353784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.353755    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.353969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.353949    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.354070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.353993    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwns\" (UniqueName: \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.430905 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.430875    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:23:08.435271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.435251    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.453539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.453515    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:23:08.453873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.453851    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-249dd4\"/\"premium-w3-249dd4-kserve-self-signed-certs\""
Oct 30 19:23:08.455231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.455210    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.455329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.455261    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwns\" (UniqueName: \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.455329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.455280    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.456112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.456093    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.457899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.457872    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.486193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.486170    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwns\" (UniqueName: \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\") pod \"maas-api-revoke-sub-051fbf61-glqg2\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.556463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556378    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.556463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556428    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.556463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556443    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.557094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556587    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.557094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556624    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpbd\" (UniqueName: \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.557094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.556764    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.609536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.609499    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658010    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658062    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658117    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658142    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658205    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.658237    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpbd\" (UniqueName: \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.659017    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.659556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.659253    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.660498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.660443    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.663886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.663867    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.666965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.666938    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.677439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.677397    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpbd\" (UniqueName: \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\") pod \"premium-w3-249dd4-kserve-698b458758-lsg6h\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") " pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.748570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.748510    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:08.940435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.940360    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"]
Oct 30 19:23:08.942819 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:23:08.942790    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a6b5f_af96_493d_bf0d_f2789fbf8115.slice/crio-625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232 WatchSource:0}: Error finding container 625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232: Status 404 returned error can't find the container with id 625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232
Oct 30 19:23:08.944892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:08.944876    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:23:09.053626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.053584    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:23:09.055792 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:23:09.055759    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3649f898_395d_48f0_9b03_f2fd1cd021d9.slice/crio-bcfdbbf7ba51af7ee3637c7ecc5bb7c2904f127ab1e0588b8ec87c2dc55d3a68 WatchSource:0}: Error finding container bcfdbbf7ba51af7ee3637c7ecc5bb7c2904f127ab1e0588b8ec87c2dc55d3a68: Status 404 returned error can't find the container with id bcfdbbf7ba51af7ee3637c7ecc5bb7c2904f127ab1e0588b8ec87c2dc55d3a68
Oct 30 19:23:09.373820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.373727    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2" event={"ID":"c84a6b5f-af96-493d-bf0d-f2789fbf8115","Type":"ContainerStarted","Data":"625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232"}
Oct 30 19:23:09.375451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.375420    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" event={"ID":"3649f898-395d-48f0-9b03-f2fd1cd021d9","Type":"ContainerStarted","Data":"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"}
Oct 30 19:23:09.375584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.375459    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" event={"ID":"3649f898-395d-48f0-9b03-f2fd1cd021d9","Type":"ContainerStarted","Data":"bcfdbbf7ba51af7ee3637c7ecc5bb7c2904f127ab1e0588b8ec87c2dc55d3a68"}
Oct 30 19:23:09.418393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.418351    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podStartSLOduration=1.418338005 podStartE2EDuration="1.418338005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:09.415668681 +0000 UTC m=+1137.451181144" watchObservedRunningTime="2026-10-30 19:23:09.418338005 +0000 UTC m=+1137.453850470"
Oct 30 19:23:09.748701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:09.748666    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:11.388302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:11.388260    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2" event={"ID":"c84a6b5f-af96-493d-bf0d-f2789fbf8115","Type":"ContainerStarted","Data":"c4b6c5d178465d8ddf6c55aa139f48afa1c67c63f93da7b49c77382a2ab6398e"}
Oct 30 19:23:11.425646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:11.425568    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2" podStartSLOduration=1.454363236 podStartE2EDuration="3.425553825s" totalImagesPullingTime="1.971190589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:11.42485177 +0000 UTC m=+1139.460364216" watchObservedRunningTime="2026-10-30 19:23:11.425553825 +0000 UTC m=+1139.461066289"
Oct 30 19:23:12.395651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:12.395620    4320 generic.go:356] "Generic (PLEG): container finished" podID="c84a6b5f-af96-493d-bf0d-f2789fbf8115" containerID="c4b6c5d178465d8ddf6c55aa139f48afa1c67c63f93da7b49c77382a2ab6398e" exitCode=0
Oct 30 19:23:12.396085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:12.395697    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2" event={"ID":"c84a6b5f-af96-493d-bf0d-f2789fbf8115","Type":"ContainerDied","Data":"c4b6c5d178465d8ddf6c55aa139f48afa1c67c63f93da7b49c77382a2ab6398e"}
Oct 30 19:23:13.547066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.547036    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:13.720260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.720190    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\") pod \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") "
Oct 30 19:23:13.720260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.720239    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\") pod \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") "
Oct 30 19:23:13.720501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.720398    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\" (UniqueName: \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\") pod \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\" (UID: \"c84a6b5f-af96-493d-bf0d-f2789fbf8115\") "
Oct 30 19:23:13.720541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.720517    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca" pod "c84a6b5f-af96-493d-bf0d-f2789fbf8115" (UID: "c84a6b5f-af96-493d-bf0d-f2789fbf8115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:13.720813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.720788    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:13.722400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.722378    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca" pod "c84a6b5f-af96-493d-bf0d-f2789fbf8115" (UID: "c84a6b5f-af96-493d-bf0d-f2789fbf8115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:13.722484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.722450    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns" pod "c84a6b5f-af96-493d-bf0d-f2789fbf8115" (UID: "c84a6b5f-af96-493d-bf0d-f2789fbf8115"). InnerVolumeSpecName "kube-api-access-zcwns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:13.822039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.822015    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwns\" (UniqueName: \"kubernetes.io/projected/c84a6b5f-af96-493d-bf0d-f2789fbf8115-kube-api-access-zcwns\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:13.822126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:13.822039    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84a6b5f-af96-493d-bf0d-f2789fbf8115-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:14.408352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:14.408316    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2" event={"ID":"c84a6b5f-af96-493d-bf0d-f2789fbf8115","Type":"ContainerDied","Data":"625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232"}
Oct 30 19:23:14.408504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:14.408368    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625b52bbb60eb93875fff48a41ec99a13e51bb50214521704f1164fce33c5232"
Oct 30 19:23:14.408504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:14.408335    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"
Oct 30 19:23:15.377011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:15.376978    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:23:18.749015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:18.748977    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:18.761303 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:18.761275    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:19.453433 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:19.453401    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:23:20.535782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.535751    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"]
Oct 30 19:23:20.536220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.536207    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c84a6b5f-af96-493d-bf0d-f2789fbf8115" containerName="revoke-keys"
Oct 30 19:23:20.536262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.536221    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84a6b5f-af96-493d-bf0d-f2789fbf8115" containerName="revoke-keys"
Oct 30 19:23:20.536314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.536305    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84a6b5f-af96-493d-bf0d-f2789fbf8115" containerName="revoke-keys"
Oct 30 19:23:20.540854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.540835    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.560569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.560532    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"]
Oct 30 19:23:20.568866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.568842    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:20.569003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.568944    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:23:20.591518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.591486    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.591680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.591646    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.591737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.591688    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v7z\" (UniqueName: \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.693016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.692988    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.693177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.693046    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.693177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.693071    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v7z\" (UniqueName: \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.693863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.693832    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.695263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.695245    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.711697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.711671    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v7z\" (UniqueName: \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\") pod \"maas-api-revoke-sub-9a7d0d52-jzgf6\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:20.858707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:20.858620    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:21.155844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:21.155823    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"]
Oct 30 19:23:21.157861 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:23:21.157825    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eab5c6_62c1_44ca_85c5_7a8585a3e52b.slice/crio-9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb WatchSource:0}: Error finding container 9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb: Status 404 returned error can't find the container with id 9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb
Oct 30 19:23:21.448688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:21.448622    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6" event={"ID":"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b","Type":"ContainerStarted","Data":"9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb"}
Oct 30 19:23:22.454295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:22.454262    4320 generic.go:356] "Generic (PLEG): container finished" podID="b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" containerID="d224c2d5736d6f33f1d2d78383a88ca3fc4b3f99febd1b1e2337dc9e0f69cc02" exitCode=0
Oct 30 19:23:22.454702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:22.454311    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6" event={"ID":"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b","Type":"ContainerDied","Data":"d224c2d5736d6f33f1d2d78383a88ca3fc4b3f99febd1b1e2337dc9e0f69cc02"}
Oct 30 19:23:23.593444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.593421    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:23.618763 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.618733    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\") pod \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") "
Oct 30 19:23:23.618933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.618917    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\") pod \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") "
Oct 30 19:23:23.619066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.618955    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\" (UniqueName: \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\") pod \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\" (UID: \"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b\") "
Oct 30 19:23:23.619066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.619021    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca" pod "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" (UID: "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:23.619323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.619305    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:23.620941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.620913    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca" pod "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" (UID: "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:23.621044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.621025    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z" pod "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" (UID: "b4eab5c6-62c1-44ca-85c5-7a8585a3e52b"). InnerVolumeSpecName "kube-api-access-m7v7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:23.720492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.720421    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:23.720492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:23.720447    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7v7z\" (UniqueName: \"kubernetes.io/projected/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b-kube-api-access-m7v7z\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:24.465001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:24.464974    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"
Oct 30 19:23:24.465164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:24.464968    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6" event={"ID":"b4eab5c6-62c1-44ca-85c5-7a8585a3e52b","Type":"ContainerDied","Data":"9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb"}
Oct 30 19:23:24.465164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:24.465095    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e29ea02ef48378242bce78ecbad00fdd5b2b2bd4e0bbf6f98ee9ed2d7f893cb"
Oct 30 19:23:25.120913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.120879    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 19:23:25.121426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.121411    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" containerName="revoke-keys"
Oct 30 19:23:25.121467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.121427    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" containerName="revoke-keys"
Oct 30 19:23:25.121535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.121523    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" containerName="revoke-keys"
Oct 30 19:23:25.125207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.125186    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.134830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.134808    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.134936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.134838    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.134936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.134863    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135043    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135104    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135194    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135279    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2x7\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135316    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.135353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.135345    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.154965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.154941    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 19:23:25.236005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.235971    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236053    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2x7\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236092    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236117    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236154    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236184    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236224    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236293    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236527 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236350    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236571    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236659    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236824    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.236934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.236860    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.237162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.237144    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.238287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.238267    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.238869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.238848    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.254750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.254715    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.255090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.255064    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2x7\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.438716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.438693    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:25.751617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.751528    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 19:23:25.754629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:23:25.754588    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda546609d_ac82_4bf7_816c_ec2e4c7ce11a.slice/crio-0c9ced4f691102242bef6d5e2657c76f5b1d87601f54c44cea9ee95eef579755 WatchSource:0}: Error finding container 0c9ced4f691102242bef6d5e2657c76f5b1d87601f54c44cea9ee95eef579755: Status 404 returned error can't find the container with id 0c9ced4f691102242bef6d5e2657c76f5b1d87601f54c44cea9ee95eef579755
Oct 30 19:23:25.757089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.757054    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:23:25.757181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.757120    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:23:25.757181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:25.757149    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:23:26.476913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:26.476876    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" event={"ID":"a546609d-ac82-4bf7-816c-ec2e4c7ce11a","Type":"ContainerStarted","Data":"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"}
Oct 30 19:23:26.477279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:26.476940    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" event={"ID":"a546609d-ac82-4bf7-816c-ec2e4c7ce11a","Type":"ContainerStarted","Data":"0c9ced4f691102242bef6d5e2657c76f5b1d87601f54c44cea9ee95eef579755"}
Oct 30 19:23:26.545939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:26.545851    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podStartSLOduration=1.545826809 podStartE2EDuration="1.545826809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:26.531727595 +0000 UTC m=+1154.567240084" watchObservedRunningTime="2026-10-30 19:23:26.545826809 +0000 UTC m=+1154.581339278"
Oct 30 19:23:27.439386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:27.439357    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:27.440858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:27.440828    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:27.440997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:27.440900    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:28.440062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:28.440025    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:28.440706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:28.440674    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:28.701078 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:23:28.700951    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:23:29.439786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:29.439751    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:29.440048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:29.439809    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:30.440041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:30.440007    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:30.440527 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:30.440067    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:31.439364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:31.439328    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:31.439541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:31.439388    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:32.439106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:32.439071    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:23:32.439734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:32.439704    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:23:33.551092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:33.551055    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:33.551609 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:33.551582    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:33.552563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:33.552538    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 19:23:41.525693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:41.525660    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:23:47.212178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.212140    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"]
Oct 30 19:23:47.219632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.219588    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.235547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.235521    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"]
Oct 30 19:23:47.244449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.244429    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:47.244576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.244449    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:23:47.296480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.296453    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.296615 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.296490    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.296671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.296618    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrj6\" (UniqueName: \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.397908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.397871    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.398055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.397935    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.398055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.398027    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrj6\" (UniqueName: \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.398729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.398628    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.400449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.400420    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.416791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.416764    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrj6\" (UniqueName: \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\") pod \"maas-api-revoke-sub-fd8a71bd-tznff\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.533577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.533496    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:47.832075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:47.832053    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"]
Oct 30 19:23:47.834050 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:23:47.834021    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133d1ec5_66c7_4be4_8981_9bfb401a6d65.slice/crio-db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c WatchSource:0}: Error finding container db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c: Status 404 returned error can't find the container with id db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c
Oct 30 19:23:48.659465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:48.659433    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff" event={"ID":"133d1ec5-66c7-4be4-8981-9bfb401a6d65","Type":"ContainerStarted","Data":"d7ae5b556510b7403515f123690dbdf1f8153d971681baa792a0cd99fd8983c6"}
Oct 30 19:23:48.659924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:48.659488    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff" event={"ID":"133d1ec5-66c7-4be4-8981-9bfb401a6d65","Type":"ContainerStarted","Data":"db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c"}
Oct 30 19:23:48.698536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:48.698492    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff" podStartSLOduration=1.698478293 podStartE2EDuration="1.698478293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:48.697322272 +0000 UTC m=+1176.732834736" watchObservedRunningTime="2026-10-30 19:23:48.698478293 +0000 UTC m=+1176.733990757"
Oct 30 19:23:49.666531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:49.666496    4320 generic.go:356] "Generic (PLEG): container finished" podID="133d1ec5-66c7-4be4-8981-9bfb401a6d65" containerID="d7ae5b556510b7403515f123690dbdf1f8153d971681baa792a0cd99fd8983c6" exitCode=0
Oct 30 19:23:49.666965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:49.666585    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff" event={"ID":"133d1ec5-66c7-4be4-8981-9bfb401a6d65","Type":"ContainerDied","Data":"d7ae5b556510b7403515f123690dbdf1f8153d971681baa792a0cd99fd8983c6"}
Oct 30 19:23:50.824454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.824428    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:50.938034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.937966    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\" (UniqueName: \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\") pod \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") "
Oct 30 19:23:50.938034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.938018    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\") pod \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") "
Oct 30 19:23:50.938254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.938117    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\") pod \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\" (UID: \"133d1ec5-66c7-4be4-8981-9bfb401a6d65\") "
Oct 30 19:23:50.938475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.938439    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca" pod "133d1ec5-66c7-4be4-8981-9bfb401a6d65" (UID: "133d1ec5-66c7-4be4-8981-9bfb401a6d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:50.940157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.940132    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca" pod "133d1ec5-66c7-4be4-8981-9bfb401a6d65" (UID: "133d1ec5-66c7-4be4-8981-9bfb401a6d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:50.940157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:50.940148    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6" pod "133d1ec5-66c7-4be4-8981-9bfb401a6d65" (UID: "133d1ec5-66c7-4be4-8981-9bfb401a6d65"). InnerVolumeSpecName "kube-api-access-qgrj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:51.039137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.039110    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:51.039137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.039137    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrj6\" (UniqueName: \"kubernetes.io/projected/133d1ec5-66c7-4be4-8981-9bfb401a6d65-kube-api-access-qgrj6\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:51.039294 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.039150    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133d1ec5-66c7-4be4-8981-9bfb401a6d65-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:23:51.679393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.679364    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"
Oct 30 19:23:51.679561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.679402    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff" event={"ID":"133d1ec5-66c7-4be4-8981-9bfb401a6d65","Type":"ContainerDied","Data":"db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c"}
Oct 30 19:23:51.679561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:51.679447    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db16cc58a6000078e1bfd447aa8113288249f3f383e26bbeaee98ca9af9e544c"
Oct 30 19:23:58.644423 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:23:58.644387    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:23:59.842950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.842922    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"]
Oct 30 19:23:59.843419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.843405    4320 cpu_manager.go:414] "containerMap: removing container" podUID="133d1ec5-66c7-4be4-8981-9bfb401a6d65" containerName="revoke-keys"
Oct 30 19:23:59.843467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.843419    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133d1ec5-66c7-4be4-8981-9bfb401a6d65" containerName="revoke-keys"
Oct 30 19:23:59.843518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.843511    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="133d1ec5-66c7-4be4-8981-9bfb401a6d65" containerName="revoke-keys"
Oct 30 19:23:59.848121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.848102    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.864922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.864894    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"]
Oct 30 19:23:59.871669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.871647    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.871802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.871689    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.871910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.871886    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sbn\" (UniqueName: \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.873965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.873944    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:59.874118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.874099    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:23:59.973246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.973221    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sbn\" (UniqueName: \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.973410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.973347    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.973410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.973385    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.974114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.974086    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.975666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.975646    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:23:59.992497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:23:59.992471    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sbn\" (UniqueName: \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\") pod \"maas-api-revoke-sub-d9bc1a7d-dbpbj\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:24:00.161637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.161610    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:24:00.458165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.458138    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"]
Oct 30 19:24:00.460989 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:00.460958    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2732b174_6f99_4ba9_aec3_e0fdc4c1936d.slice/crio-2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4 WatchSource:0}: Error finding container 2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4: Status 404 returned error can't find the container with id 2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4
Oct 30 19:24:00.667633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.666316    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 19:24:00.674935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.674908    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.688033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.688001    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 19:24:00.696131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.696108    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac614-openshift-default-dockercfg-2npc8\""
Oct 30 19:24:00.742328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.742257    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj" event={"ID":"2732b174-6f99-4ba9-aec3-e0fdc4c1936d","Type":"ContainerStarted","Data":"2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4"}
Oct 30 19:24:00.782674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782649    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.782801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782683    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.782801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782715    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.782801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45x88\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.782936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782870    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.782936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782930    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.783011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.782999    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.783115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.783102    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.783154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.783144    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884483    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884547    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884582    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884628    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884659    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884698    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884722    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45x88\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884879    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884907    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.884951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.884945    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.885440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.885027    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.885440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.885153    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.885440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.885304    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.885440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.885417    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.886845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.886820    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.887038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.887024    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.903867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.903844    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:00.903961 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:00.903906    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45x88\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:01.001966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.001901    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:01.319186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.319099    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 19:24:01.349859 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:01.349829    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b59786_be29_4f1c_81d5_485eae4fe8a1.slice/crio-33ecc072abad039632809ba4f2f434b291dda749cdb57f5d0e2aaf4210f0ed4f WatchSource:0}: Error finding container 33ecc072abad039632809ba4f2f434b291dda749cdb57f5d0e2aaf4210f0ed4f: Status 404 returned error can't find the container with id 33ecc072abad039632809ba4f2f434b291dda749cdb57f5d0e2aaf4210f0ed4f
Oct 30 19:24:01.352495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.352455    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:01.352627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.352546    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:01.352627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.352592    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:01.750606 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.750570    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" event={"ID":"a2b59786-be29-4f1c-81d5-485eae4fe8a1","Type":"ContainerStarted","Data":"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"}
Oct 30 19:24:01.750816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.750645    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" event={"ID":"a2b59786-be29-4f1c-81d5-485eae4fe8a1","Type":"ContainerStarted","Data":"33ecc072abad039632809ba4f2f434b291dda749cdb57f5d0e2aaf4210f0ed4f"}
Oct 30 19:24:01.752275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.752247    4320 generic.go:356] "Generic (PLEG): container finished" podID="2732b174-6f99-4ba9-aec3-e0fdc4c1936d" containerID="bb21743c6019f04a99482abf6df5c6926358ba903ea6a3bf8beacda729ead83e" exitCode=0
Oct 30 19:24:01.752396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.752315    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj" event={"ID":"2732b174-6f99-4ba9-aec3-e0fdc4c1936d","Type":"ContainerDied","Data":"bb21743c6019f04a99482abf6df5c6926358ba903ea6a3bf8beacda729ead83e"}
Oct 30 19:24:01.796876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:01.796826    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podStartSLOduration=1.7968085120000001 podStartE2EDuration="1.796808512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:01.79353738 +0000 UTC m=+1189.829049844" watchObservedRunningTime="2026-10-30 19:24:01.796808512 +0000 UTC m=+1189.832320980"
Oct 30 19:24:02.002243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:02.002164    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:02.004115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:02.004086    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:02.004241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:02.004141    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:02.931194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:02.931171    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:24:03.002380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.002348    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:03.002898 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.002408    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:03.021946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.021875    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\") pod \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") "
Oct 30 19:24:03.022056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.021967    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\" (UniqueName: \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\") pod \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") "
Oct 30 19:24:03.022056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.022005    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\") pod \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\" (UID: \"2732b174-6f99-4ba9-aec3-e0fdc4c1936d\") "
Oct 30 19:24:03.022326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.022297    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca" pod "2732b174-6f99-4ba9-aec3-e0fdc4c1936d" (UID: "2732b174-6f99-4ba9-aec3-e0fdc4c1936d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:03.022573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.022551    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:03.024057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.024035    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca" pod "2732b174-6f99-4ba9-aec3-e0fdc4c1936d" (UID: "2732b174-6f99-4ba9-aec3-e0fdc4c1936d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:03.024144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.024110    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn" pod "2732b174-6f99-4ba9-aec3-e0fdc4c1936d" (UID: "2732b174-6f99-4ba9-aec3-e0fdc4c1936d"). InnerVolumeSpecName "kube-api-access-w2sbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:03.123971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.123942    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sbn\" (UniqueName: \"kubernetes.io/projected/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-kube-api-access-w2sbn\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:03.123971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.123968    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2732b174-6f99-4ba9-aec3-e0fdc4c1936d-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:03.764690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.764662    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"
Oct 30 19:24:03.764891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.764709    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj" event={"ID":"2732b174-6f99-4ba9-aec3-e0fdc4c1936d","Type":"ContainerDied","Data":"2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4"}
Oct 30 19:24:03.764891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.764762    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa6993637d6dee354b17c83fa8b3049d5235caf575423622cb3f7a8d969b0b4"
Oct 30 19:24:03.940411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:03.940379    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:24:04.002871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:04.002843    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:04.003260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:04.002889    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:05.002187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:05.002152    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:05.002920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:05.002886    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:06.002592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:06.002550    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:06.002813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:06.002629    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:07.002590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:07.002548    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:07.003211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:07.003177    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:08.003151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:08.003112    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:24:08.003529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:08.003182    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:24:09.118930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:09.118897    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:09.119883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:09.119857    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:09.122134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:09.122106    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 19:24:11.728542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.728511    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"]
Oct 30 19:24:11.729052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.729037    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2732b174-6f99-4ba9-aec3-e0fdc4c1936d" containerName="revoke-keys"
Oct 30 19:24:11.729096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.729053    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2732b174-6f99-4ba9-aec3-e0fdc4c1936d" containerName="revoke-keys"
Oct 30 19:24:11.729160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.729151    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2732b174-6f99-4ba9-aec3-e0fdc4c1936d" containerName="revoke-keys"
Oct 30 19:24:11.732762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.732737    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.760556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.760532    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:11.760712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.760573    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:24:11.763626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.763577    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"]
Oct 30 19:24:11.863137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.863099    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvz77\" (UniqueName: \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.863288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.863255    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.863288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.863282    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.964544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.964515    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.964544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.964546    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.964772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.964629    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvz77\" (UniqueName: \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.965438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.965411    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.966958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.966937    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:11.983027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:11.982971    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvz77\" (UniqueName: \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\") pod \"maas-api-revoke-sub-2603c1eb-wzbm6\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:12.045747 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:12.045722    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:12.344765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:12.344734    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"]
Oct 30 19:24:12.346922 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:12.346892    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383f2689_f556_4951_b3ec_5394b5273267.slice/crio-7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d WatchSource:0}: Error finding container 7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d: Status 404 returned error can't find the container with id 7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d
Oct 30 19:24:12.844190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:12.844156    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6" event={"ID":"383f2689-f556-4951-b3ec-5394b5273267","Type":"ContainerStarted","Data":"6afd8b5748ab5a78fc95e4a283a6c47ecf904c08fc11a41295cbc43533932dd3"}
Oct 30 19:24:12.844518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:12.844217    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6" event={"ID":"383f2689-f556-4951-b3ec-5394b5273267","Type":"ContainerStarted","Data":"7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d"}
Oct 30 19:24:12.907106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:12.907062    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6" podStartSLOduration=1.9070458449999999 podStartE2EDuration="1.907045845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:12.902069967 +0000 UTC m=+1200.937582431" watchObservedRunningTime="2026-10-30 19:24:12.907045845 +0000 UTC m=+1200.942558310"
Oct 30 19:24:13.852778 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:13.852738    4320 generic.go:356] "Generic (PLEG): container finished" podID="383f2689-f556-4951-b3ec-5394b5273267" containerID="6afd8b5748ab5a78fc95e4a283a6c47ecf904c08fc11a41295cbc43533932dd3" exitCode=0
Oct 30 19:24:13.853212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:13.852830    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6" event={"ID":"383f2689-f556-4951-b3ec-5394b5273267","Type":"ContainerDied","Data":"6afd8b5748ab5a78fc95e4a283a6c47ecf904c08fc11a41295cbc43533932dd3"}
Oct 30 19:24:14.020770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.020726    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 19:24:14.025931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.025910    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.040458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.040396    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"distinct-w2-9efc72-kserve-self-signed-certs\""
Oct 30 19:24:14.043631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.043590    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 19:24:14.088721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.088691    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.088935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.088760    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jfg\" (UniqueName: \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.089242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.089217    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.089346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.089297    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.089434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.089418    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.089497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.089480    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.190979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.190947    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191016    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191053    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191172    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191221    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jfg\" (UniqueName: \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191280    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191391    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191501    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.191697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.191679    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.210618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.193667    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.211362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.211334    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.236287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.236254    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jfg\" (UniqueName: \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\") pod \"distinct-w2-9efc72-kserve-58fddb74b7-xxdpx\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.341044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.341012    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:14.643921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.643886    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 19:24:14.647504 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:14.647479    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfec8e3f9_0c45_4a84_b703_430f061ba2f2.slice/crio-cfde0de1af74919bc73917b97f923e8bd9b73f79f77e7a6ba77845336ee6518e WatchSource:0}: Error finding container cfde0de1af74919bc73917b97f923e8bd9b73f79f77e7a6ba77845336ee6518e: Status 404 returned error can't find the container with id cfde0de1af74919bc73917b97f923e8bd9b73f79f77e7a6ba77845336ee6518e
Oct 30 19:24:14.859816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.859780    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" event={"ID":"fec8e3f9-0c45-4a84-b703-430f061ba2f2","Type":"ContainerStarted","Data":"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"}
Oct 30 19:24:14.860211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.859829    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" event={"ID":"fec8e3f9-0c45-4a84-b703-430f061ba2f2","Type":"ContainerStarted","Data":"cfde0de1af74919bc73917b97f923e8bd9b73f79f77e7a6ba77845336ee6518e"}
Oct 30 19:24:14.901704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.900377    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podStartSLOduration=1.900355118 podStartE2EDuration="1.900355118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:14.899048913 +0000 UTC m=+1202.934561377" watchObservedRunningTime="2026-10-30 19:24:14.900355118 +0000 UTC m=+1202.935867587"
Oct 30 19:24:14.993449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:14.993416    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:15.104489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.104455    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\") pod \"383f2689-f556-4951-b3ec-5394b5273267\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") "
Oct 30 19:24:15.104680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.104502    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\") pod \"383f2689-f556-4951-b3ec-5394b5273267\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") "
Oct 30 19:24:15.104680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.104564    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\" (UniqueName: \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\") pod \"383f2689-f556-4951-b3ec-5394b5273267\" (UID: \"383f2689-f556-4951-b3ec-5394b5273267\") "
Oct 30 19:24:15.104982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.104957    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca" pod "383f2689-f556-4951-b3ec-5394b5273267" (UID: "383f2689-f556-4951-b3ec-5394b5273267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:15.106682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.106659    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77" pod "383f2689-f556-4951-b3ec-5394b5273267" (UID: "383f2689-f556-4951-b3ec-5394b5273267"). InnerVolumeSpecName "kube-api-access-vvz77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:15.107067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.107045    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca" pod "383f2689-f556-4951-b3ec-5394b5273267" (UID: "383f2689-f556-4951-b3ec-5394b5273267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:15.206660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.206628    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/383f2689-f556-4951-b3ec-5394b5273267-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:15.206660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.206657    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/383f2689-f556-4951-b3ec-5394b5273267-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:15.206814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.206672    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvz77\" (UniqueName: \"kubernetes.io/projected/383f2689-f556-4951-b3ec-5394b5273267-kube-api-access-vvz77\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:15.342119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.342087    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:15.874060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.874032    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"
Oct 30 19:24:15.874719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.874066    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6" event={"ID":"383f2689-f556-4951-b3ec-5394b5273267","Type":"ContainerDied","Data":"7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d"}
Oct 30 19:24:15.874827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:15.874780    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7596c4dfb9f4de1fbf4481a14e09623a7cd1fd27ac22c73df0e1a6e195e4b84d"
Oct 30 19:24:18.807172 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.807144    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 19:24:18.807677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.807661    4320 cpu_manager.go:414] "containerMap: removing container" podUID="383f2689-f556-4951-b3ec-5394b5273267" containerName="revoke-keys"
Oct 30 19:24:18.807735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.807678    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383f2689-f556-4951-b3ec-5394b5273267" containerName="revoke-keys"
Oct 30 19:24:18.807787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.807778    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="383f2689-f556-4951-b3ec-5394b5273267" containerName="revoke-keys"
Oct 30 19:24:18.811263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.811247    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.848430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.848401    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 19:24:18.964087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964056    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964108    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964161    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964325 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964262    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964373    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964454    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964543    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964592    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb49w\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:18.964693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:18.964679    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.065930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.065864    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.065930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.065906    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb49w\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.065953    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.065991    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066045    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066078    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066105    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066170    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066282    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066422    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066469    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066590    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066671    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.066906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.066884    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.068224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.068197    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.068537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.068520    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.084378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.084355    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.084572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.084553    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb49w\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\") pod \"maas-default-gateway-openshift-default-6f54779779-8k46t\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.126138 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.126116    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:19.443819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.443792    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 19:24:19.445837 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:19.445811    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a12a296_1061_4e61_8ff8_c6a63a5320da.slice/crio-8b546c82e82a053bf21aeb024d8b7a41352301101e5d6b21911d02f75a8f094b WatchSource:0}: Error finding container 8b546c82e82a053bf21aeb024d8b7a41352301101e5d6b21911d02f75a8f094b: Status 404 returned error can't find the container with id 8b546c82e82a053bf21aeb024d8b7a41352301101e5d6b21911d02f75a8f094b
Oct 30 19:24:19.448825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.448786    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:19.448914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.448855    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:19.448914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.448887    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:19.909614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.909565    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" event={"ID":"7a12a296-1061-4e61-8ff8-c6a63a5320da","Type":"ContainerStarted","Data":"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"}
Oct 30 19:24:19.910044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.909720    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" event={"ID":"7a12a296-1061-4e61-8ff8-c6a63a5320da","Type":"ContainerStarted","Data":"8b546c82e82a053bf21aeb024d8b7a41352301101e5d6b21911d02f75a8f094b"}
Oct 30 19:24:19.951876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:19.951825    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podStartSLOduration=1.9518113430000001 podStartE2EDuration="1.951811343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:19.948818877 +0000 UTC m=+1207.984331341" watchObservedRunningTime="2026-10-30 19:24:19.951811343 +0000 UTC m=+1207.987323806"
Oct 30 19:24:20.127136 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:20.127105    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:20.129051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:20.129025    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:20.129196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:20.129072    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:21.126774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:21.126742    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:21.127161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:21.126793    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:22.126744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:22.126714    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:22.126911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:22.126759    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:23.126704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:23.126670    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:23.126880 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:23.126725    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:24.126874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.126832    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:24.127356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.126912    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:24.341582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.341548    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:24.354610 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.354580    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:24.428706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.428671    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"]
Oct 30 19:24:24.432985 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.432968    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.442392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.442364    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.442512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.442457    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrflt\" (UniqueName: \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.442673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.442650    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.447794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.447776    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:24.448106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.448083    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:24:24.451900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.451868    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"]
Oct 30 19:24:24.543664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.543592    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.543820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.543709    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrflt\" (UniqueName: \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.543820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.543793    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.544418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.544391    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.546001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.545977    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.565442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.565419    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrflt\" (UniqueName: \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\") pod \"maas-api-revoke-sub-cfe10893-m556k\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.748184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.748111    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:24.948729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:24.948655    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:24:25.095714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.095652    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"]
Oct 30 19:24:25.127422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.127054    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:25.127956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.127464    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:25.165143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.165120    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 19:24:25.180853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.180831    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.199570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.199528    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b97379-openshift-default-dockercfg-zcgdk\""
Oct 30 19:24:25.201340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.201267    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 19:24:25.253180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253150    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253209    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nkr\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253242    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253261    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253291    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253315    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253347    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253379    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.253445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.253404    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354042    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354089    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354141    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354193    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354235    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354323    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354402    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nkr\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354532    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354561    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354632    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354722    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354892    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.354958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.354942    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.355170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.355155    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.356797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.356769    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.356900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.356889    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.396014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.395707    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.396014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.395949    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nkr\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.522757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.522723    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:25.843147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.843118    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 19:24:25.851155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:25.851122    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd34e558_1afb_4b30_a736_f7c9f7f96ac3.slice/crio-a97520926cbc8aed5c73b7991d6d0bc355be11d8f6ebcde3715a06d8f2fbd389 WatchSource:0}: Error finding container a97520926cbc8aed5c73b7991d6d0bc355be11d8f6ebcde3715a06d8f2fbd389: Status 404 returned error can't find the container with id a97520926cbc8aed5c73b7991d6d0bc355be11d8f6ebcde3715a06d8f2fbd389
Oct 30 19:24:25.853720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.853684    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:25.853808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.853768    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:25.853912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.853814    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:25.944789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.944758    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k" event={"ID":"f2e9d119-6d77-4b77-82fb-52176623f38e","Type":"ContainerStarted","Data":"958c730c937d6bf5f6402f2de4c5d8a7dd4111aff1c515a6e14177a2dd551c7f"}
Oct 30 19:24:25.944981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.944954    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k" event={"ID":"f2e9d119-6d77-4b77-82fb-52176623f38e","Type":"ContainerStarted","Data":"bfbd4c46ac94b32c2d4639f31310373477a1881aa35029bacfb7ab3a30258ccc"}
Oct 30 19:24:25.946785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.946627    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" event={"ID":"cd34e558-1afb-4b30-a736-f7c9f7f96ac3","Type":"ContainerStarted","Data":"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"}
Oct 30 19:24:25.946785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.946680    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" event={"ID":"cd34e558-1afb-4b30-a736-f7c9f7f96ac3","Type":"ContainerStarted","Data":"a97520926cbc8aed5c73b7991d6d0bc355be11d8f6ebcde3715a06d8f2fbd389"}
Oct 30 19:24:25.983191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:25.982052    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k" podStartSLOduration=1.9820348399999999 podStartE2EDuration="1.98203484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:25.981408486 +0000 UTC m=+1214.016920951" watchObservedRunningTime="2026-10-30 19:24:25.98203484 +0000 UTC m=+1214.017547310"
Oct 30 19:24:26.030021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.029965    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podStartSLOduration=1.029950622 podStartE2EDuration="1.029950622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:26.027753307 +0000 UTC m=+1214.063265770" watchObservedRunningTime="2026-10-30 19:24:26.029950622 +0000 UTC m=+1214.065463086"
Oct 30 19:24:26.126704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.126626    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:26.126704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.126676    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:26.523413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.523375    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:26.524921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.524898    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:26.525039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.524942    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:26.964483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.964439    4320 generic.go:356] "Generic (PLEG): container finished" podID="f2e9d119-6d77-4b77-82fb-52176623f38e" containerID="958c730c937d6bf5f6402f2de4c5d8a7dd4111aff1c515a6e14177a2dd551c7f" exitCode=0
Oct 30 19:24:26.964718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:26.964522    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k" event={"ID":"f2e9d119-6d77-4b77-82fb-52176623f38e","Type":"ContainerDied","Data":"958c730c937d6bf5f6402f2de4c5d8a7dd4111aff1c515a6e14177a2dd551c7f"}
Oct 30 19:24:27.127450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:27.127417    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:24:27.127624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:27.127472    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:24:27.523478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:27.523436    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:27.523971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:27.523496    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:28.215070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.215038    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:28.252027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.251963    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:28.252448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.252417    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:28.253375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.253355    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 19:24:28.312185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.312147    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\") pod \"f2e9d119-6d77-4b77-82fb-52176623f38e\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") "
Oct 30 19:24:28.312366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.312289    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\" (UniqueName: \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\") pod \"f2e9d119-6d77-4b77-82fb-52176623f38e\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") "
Oct 30 19:24:28.312366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.312353    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\") pod \"f2e9d119-6d77-4b77-82fb-52176623f38e\" (UID: \"f2e9d119-6d77-4b77-82fb-52176623f38e\") "
Oct 30 19:24:28.313723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.313694    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca" pod "f2e9d119-6d77-4b77-82fb-52176623f38e" (UID: "f2e9d119-6d77-4b77-82fb-52176623f38e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:28.316652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.316622    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca" pod "f2e9d119-6d77-4b77-82fb-52176623f38e" (UID: "f2e9d119-6d77-4b77-82fb-52176623f38e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:28.316730 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.316693    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt" pod "f2e9d119-6d77-4b77-82fb-52176623f38e" (UID: "f2e9d119-6d77-4b77-82fb-52176623f38e"). InnerVolumeSpecName "kube-api-access-wrflt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:28.414446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.414412    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrflt\" (UniqueName: \"kubernetes.io/projected/f2e9d119-6d77-4b77-82fb-52176623f38e-kube-api-access-wrflt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:28.414446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.414444    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:28.414653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.414458    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e9d119-6d77-4b77-82fb-52176623f38e-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:24:28.524041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.523961    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:28.524041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:28.524010    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:28.644129 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:24:28.644083    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:24:29.005648 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:29.005568    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"
Oct 30 19:24:29.005847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:29.005710    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k" event={"ID":"f2e9d119-6d77-4b77-82fb-52176623f38e","Type":"ContainerDied","Data":"bfbd4c46ac94b32c2d4639f31310373477a1881aa35029bacfb7ab3a30258ccc"}
Oct 30 19:24:29.005847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:29.005773    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfbd4c46ac94b32c2d4639f31310373477a1881aa35029bacfb7ab3a30258ccc"
Oct 30 19:24:29.523022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:29.522986    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:29.523277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:29.523036    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:30.320878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.320840    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 19:24:30.321562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.321541    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f2e9d119-6d77-4b77-82fb-52176623f38e" containerName="revoke-keys"
Oct 30 19:24:30.321562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.321560    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e9d119-6d77-4b77-82fb-52176623f38e" containerName="revoke-keys"
Oct 30 19:24:30.321766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.321663    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e9d119-6d77-4b77-82fb-52176623f38e" containerName="revoke-keys"
Oct 30 19:24:30.325218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.325198    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.356801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.356776    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e67b84-openshift-default-dockercfg-dvgvb\""
Oct 30 19:24:30.359262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.359236    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 19:24:30.438146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438109    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438151    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438228    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmhg\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438271    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438309    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438353    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438411    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438438    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.438582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.438505    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.523529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.523492    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:30.523744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.523555    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:30.540151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540121    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540163    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540202    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmhg\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540243    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540303    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540524 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540469    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540524 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540513    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540539    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540578    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540622    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.540826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.540697    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.541172 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.541144    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.541293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.541188    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.541466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.541446    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.542952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.542925    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.543152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.543136    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.558714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.558694    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.559040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.559021    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmhg\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.638832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.638806    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:30.658324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:30.658188    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:24:31.002552 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.001007    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 19:24:31.013455 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:31.013423    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd988974a_2335_4851_bb94_d4964feecb3d.slice/crio-1feaf6eaf7e107d3aed14261d8c11c4638b39c405141746cb05289fd66e540b9 WatchSource:0}: Error finding container 1feaf6eaf7e107d3aed14261d8c11c4638b39c405141746cb05289fd66e540b9: Status 404 returned error can't find the container with id 1feaf6eaf7e107d3aed14261d8c11c4638b39c405141746cb05289fd66e540b9
Oct 30 19:24:31.016874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.016478    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:31.016969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.016944    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:31.017014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.017000    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:31.523885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.523846    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:31.524360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:31.523896    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:32.038213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.038171    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" event={"ID":"d988974a-2335-4851-bb94-d4964feecb3d","Type":"ContainerStarted","Data":"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"}
Oct 30 19:24:32.039567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.039539    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" event={"ID":"d988974a-2335-4851-bb94-d4964feecb3d","Type":"ContainerStarted","Data":"1feaf6eaf7e107d3aed14261d8c11c4638b39c405141746cb05289fd66e540b9"}
Oct 30 19:24:32.120706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.120650    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podStartSLOduration=2.120629157 podStartE2EDuration="2.120629157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:32.098059719 +0000 UTC m=+1220.133572206" watchObservedRunningTime="2026-10-30 19:24:32.120629157 +0000 UTC m=+1220.156141613"
Oct 30 19:24:32.524107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.524055    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:24:32.524639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.524129    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 19:24:32.639689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.639647    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:32.641764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.641685    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:32.641764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:32.641746    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:33.640474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:33.640102    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:33.640474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:33.640166    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:33.749403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:33.749369    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:33.749818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:33.749798    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:33.750627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:33.750588    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 19:24:34.639950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:34.639919    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:34.640158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:34.639971    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:35.640252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:35.640218    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:35.640653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:35.640269    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:36.640022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:36.639990    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:36.640189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:36.640050    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:37.639964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:37.639929    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:24:37.640348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:37.640003    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:24:38.644099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:38.644070    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:38.644490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:38.644360    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:38.644998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:38.644978    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 19:24:43.897341 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:43.897295    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:45.594185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.594150    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:24:45.598197 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.598179    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.612420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.612401    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6e0c55-openshift-default-dockercfg-mndx2\""
Oct 30 19:24:45.627675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.627653    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:24:45.724762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724730    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.724875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.724875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724851    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.724949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724908    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crj94\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.724949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724937    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.725017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.724968    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.725051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.725013    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.725123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.725110    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.725161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.725131    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.825935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.825904    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826537    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826685    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826721    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826805    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826865    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.826944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826901    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.827471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.827021    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crj94\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.827471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.827070    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.827687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.826482    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.828058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.827919    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.828532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.828504    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.828676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.828509    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.829032 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.828776    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.831469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.831446    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.831987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.831966    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.850045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.849815    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.850045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.849993    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crj94\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:45.911851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:45.911829    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:46.238662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.238637    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:24:46.241088 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:46.241060    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e9348a_9635_4008_85af_6742197e476f.slice/crio-5afb2ee90395629585a85562e0f80c11757f3eb5060d3b4d8c3ccd29d05e031e WatchSource:0}: Error finding container 5afb2ee90395629585a85562e0f80c11757f3eb5060d3b4d8c3ccd29d05e031e: Status 404 returned error can't find the container with id 5afb2ee90395629585a85562e0f80c11757f3eb5060d3b4d8c3ccd29d05e031e
Oct 30 19:24:46.243569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.243530    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:46.243692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.243627    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:46.243692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.243674    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:46.445634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.445578    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 19:24:46.458948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.458918    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.473366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.473342    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9efc72\"/\"distinct-2-w2-9efc72-kserve-self-signed-certs\""
Oct 30 19:24:46.476208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.476184    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 19:24:46.638055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638019    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzvg\" (UniqueName: \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.638523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638242    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.638523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638336    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.638708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638681    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.638770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638740    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.638827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.638766    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.739852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.739815    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.739861    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.739983    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzvg\" (UniqueName: \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740073    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740131    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740270 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740203    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740408    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740533    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.740684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.740608    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.742715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.742693    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.743213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.743190    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.758835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.758811    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzvg\" (UniqueName: \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\") pod \"distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") " pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:46.778719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:46.778695    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:47.098763 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:47.098733    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828a1b40_db68_403d_a73d_e8ffc6b31e8f.slice/crio-caaae1b8998417583a29ad3d2c5a23e15c10efe185cc8d164254ea380ea4f708 WatchSource:0}: Error finding container caaae1b8998417583a29ad3d2c5a23e15c10efe185cc8d164254ea380ea4f708: Status 404 returned error can't find the container with id caaae1b8998417583a29ad3d2c5a23e15c10efe185cc8d164254ea380ea4f708
Oct 30 19:24:47.099364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.099324    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 19:24:47.155546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.154965    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" event={"ID":"e4e9348a-9635-4008-85af-6742197e476f","Type":"ContainerStarted","Data":"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"}
Oct 30 19:24:47.155546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.155017    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" event={"ID":"e4e9348a-9635-4008-85af-6742197e476f","Type":"ContainerStarted","Data":"5afb2ee90395629585a85562e0f80c11757f3eb5060d3b4d8c3ccd29d05e031e"}
Oct 30 19:24:47.157170 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.157141    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" event={"ID":"828a1b40-db68-403d-a73d-e8ffc6b31e8f","Type":"ContainerStarted","Data":"caaae1b8998417583a29ad3d2c5a23e15c10efe185cc8d164254ea380ea4f708"}
Oct 30 19:24:47.234160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.234099    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podStartSLOduration=2.234077676 podStartE2EDuration="2.234077676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:47.231391367 +0000 UTC m=+1235.266903844" watchObservedRunningTime="2026-10-30 19:24:47.234077676 +0000 UTC m=+1235.269590141"
Oct 30 19:24:47.912936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.912904    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:47.914400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.914377    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:24:47.914512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:47.914420    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:24:48.164919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:48.164836    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" event={"ID":"828a1b40-db68-403d-a73d-e8ffc6b31e8f","Type":"ContainerStarted","Data":"fbc3b0c8c5968e0905406b33d0c40b5bf60697ce94c7a54716062bcba2269fd6"}
Oct 30 19:24:48.224748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:48.224687    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podStartSLOduration=2.224673009 podStartE2EDuration="2.224673009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:48.222321483 +0000 UTC m=+1236.257833949" watchObservedRunningTime="2026-10-30 19:24:48.224673009 +0000 UTC m=+1236.260185519"
Oct 30 19:24:48.790903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:48.790760    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:48.912717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:48.912678    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:24:48.912873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:48.912751    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:24:49.912370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:49.912318    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:24:49.912899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:49.912388    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:24:50.912755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:50.912716    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:24:50.913137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:50.912780    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:24:51.913005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:51.912975    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:24:51.913436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:51.913025    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:24:53.171745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:53.171707    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:53.172238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:53.172174    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:53.173489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:53.173466    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:24:56.779680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:56.779646    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:56.792347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:56.792320    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:57.263479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:57.263445    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:24:57.609217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:57.609130    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:24:58.646956 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:24:58.646919    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:24:59.143628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.143437    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 19:24:59.158873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.158846    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.172485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.172447    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 19:24:59.185368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.185336    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"openshift-service-ca.crt\""
Oct 30 19:24:59.196725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.196697    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"kube-root-ca.crt\""
Oct 30 19:24:59.196890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.196866    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"default-dockercfg-k65n6\""
Oct 30 19:24:59.197083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.196882    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b97379\"/\"distinct-w4-b97379-kserve-self-signed-certs\""
Oct 30 19:24:59.220621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.220578    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.220808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.220781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.220947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.220919    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.221022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.221009    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjnt\" (UniqueName: \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.221082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.221054    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.221155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.221137    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322138    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjnt\" (UniqueName: \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322468    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322556    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322591    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322739    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.322833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.322821    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.326218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.326170    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.326488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.326460    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.335568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.335545    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.335855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.335837    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.343774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.342490    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjnt\" (UniqueName: \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.343774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.342519    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\") pod \"distinct-w4-b97379-kserve-99fb485fd-tljsh\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") " pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.492762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.492645    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:24:59.808246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:24:59.807936    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 19:24:59.810990 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:24:59.810960    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183ed1f5_6b56_4b8e_8537_50a952b89d24.slice/crio-f233c8670cc1e68989c345dda2b8b358200250a87da661e68719e3842c651642 WatchSource:0}: Error finding container f233c8670cc1e68989c345dda2b8b358200250a87da661e68719e3842c651642: Status 404 returned error can't find the container with id f233c8670cc1e68989c345dda2b8b358200250a87da661e68719e3842c651642
Oct 30 19:25:00.265420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.265390    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" event={"ID":"183ed1f5-6b56-4b8e-8537-50a952b89d24","Type":"ContainerStarted","Data":"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"}
Oct 30 19:25:00.265584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.265426    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" event={"ID":"183ed1f5-6b56-4b8e-8537-50a952b89d24","Type":"ContainerStarted","Data":"f233c8670cc1e68989c345dda2b8b358200250a87da661e68719e3842c651642"}
Oct 30 19:25:00.303652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.303577    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podStartSLOduration=1.303561359 podStartE2EDuration="1.303561359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:00.302785338 +0000 UTC m=+1248.338297803" watchObservedRunningTime="2026-10-30 19:25:00.303561359 +0000 UTC m=+1248.339073837"
Oct 30 19:25:00.493151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.493116    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:25:00.838701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.838665    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:25:00.843118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.843090    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.875257 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.875225    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:25:00.943921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.943887    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944003    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944066    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944113    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944196    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944273    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bs4f\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944386    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944415    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:00.944457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:00.944449    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045394    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045480    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045513    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045555    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045617    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045697    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bs4f\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045784    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045806    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045834    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.045964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.045913    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.046115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.046004    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.046185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.046162    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.046371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.046343    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.046371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.046356    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.048420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.048392    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.048512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.048415    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.065164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.065137    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.065259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.065188    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bs4f\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.159254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.159233    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:01.475521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.475493    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:25:01.477931 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:25:01.477902    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f4bb69_de05_448b_9657_a91d2f8ce53b.slice/crio-930795e01ee731d8345b4487308e2b9076aa5f14bdf5dfb712a79cb5e5a1887b WatchSource:0}: Error finding container 930795e01ee731d8345b4487308e2b9076aa5f14bdf5dfb712a79cb5e5a1887b: Status 404 returned error can't find the container with id 930795e01ee731d8345b4487308e2b9076aa5f14bdf5dfb712a79cb5e5a1887b
Oct 30 19:25:01.480360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.480328    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:01.480423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.480394    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:01.480489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:01.480422    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:02.298886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:02.298849    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" event={"ID":"96f4bb69-de05-448b-9657-a91d2f8ce53b","Type":"ContainerStarted","Data":"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"}
Oct 30 19:25:02.299512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:02.299474    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" event={"ID":"96f4bb69-de05-448b-9657-a91d2f8ce53b","Type":"ContainerStarted","Data":"930795e01ee731d8345b4487308e2b9076aa5f14bdf5dfb712a79cb5e5a1887b"}
Oct 30 19:25:02.346957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:02.346910    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podStartSLOduration=2.346897177 podStartE2EDuration="2.346897177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:02.343514323 +0000 UTC m=+1250.379026788" watchObservedRunningTime="2026-10-30 19:25:02.346897177 +0000 UTC m=+1250.382409640"
Oct 30 19:25:03.160102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:03.160074    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:03.161693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:03.161665    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:25:03.161802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:03.161716    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:25:04.160228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:04.160195    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:25:04.160736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:04.160264    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:25:05.160210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:05.160177    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:25:05.160430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:05.160232    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:25:06.159674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:06.159640    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:25:06.159932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:06.159690    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:25:07.160121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:07.160089    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:25:07.160502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:07.160139    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:25:08.165826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:08.165796    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:08.166279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:08.166257    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:08.167050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:08.167030    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:25:09.499624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:09.498669    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:25:09.527344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:09.527316    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:25:10.358443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:10.358417    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:25:17.967715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:17.967681    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:25:28.646129 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:25:28.646087    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:25:30.337900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.337869    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:25:30.343556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.343534    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.373368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.373340    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:25:30.441760 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.441716    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.441948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.441769    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.441948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.441892    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.441953    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.442045    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.442112    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.442174    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.442264    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp7xf\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.442319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.442304    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.505966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.505932    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:25:30.510551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.510524    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.534804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.534765    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:25:30.543691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543659    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543710    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543765    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543814    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543854    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.543925    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.544001    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.544060    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp7xf\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.544105    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.544579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.544559    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.545113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.545089    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.545578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.545558    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.545811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.545795    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.545990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.545976    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.548852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.548827    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.549017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.548995    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.570474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.570437    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp7xf\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.570914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.570888    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.645123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645096    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645126    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645146    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645278    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645322    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645392    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzllh\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645419    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645458    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.645583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.645568    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.660269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.660247    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:30.746475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746446    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746518    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746537    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746555    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746651    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746697    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746728    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzllh\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746748    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.746768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.746763    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.747646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.747267    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.747646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.747274    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.747646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.747576    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.747646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.747586    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.747958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.747937    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.749462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.749442    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.749522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.749470    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.766361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.766337    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzllh\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.766519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.766501    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.829424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.829405    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:30.978041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.978019    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:25:30.978752 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:25:30.978728    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61456446_5707_4a7d_8f0d_bc34fe251d44.slice/crio-460842d646562bfdff98cde4313da0a9d1725a4971b1422b23197696ee5c37be WatchSource:0}: Error finding container 460842d646562bfdff98cde4313da0a9d1725a4971b1422b23197696ee5c37be: Status 404 returned error can't find the container with id 460842d646562bfdff98cde4313da0a9d1725a4971b1422b23197696ee5c37be
Oct 30 19:25:30.981204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.981167    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:30.981332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.981248    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:30.981332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:30.981294    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:31.133970 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:25:31.133940    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c71e168_4657_48f9_bc1b_d2b6446ddfaf.slice/crio-6ab507648ce12be648009ee934ef0a972fa7fc27d1b0d574ba300843033cbd77 WatchSource:0}: Error finding container 6ab507648ce12be648009ee934ef0a972fa7fc27d1b0d574ba300843033cbd77: Status 404 returned error can't find the container with id 6ab507648ce12be648009ee934ef0a972fa7fc27d1b0d574ba300843033cbd77
Oct 30 19:25:31.134268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.134241    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:25:31.136804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.136769    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:31.136923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.136848    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:31.136923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.136893    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:25:31.526006 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.525965    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" event={"ID":"7c71e168-4657-48f9-bc1b-d2b6446ddfaf","Type":"ContainerStarted","Data":"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"}
Oct 30 19:25:31.526496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.526058    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" event={"ID":"7c71e168-4657-48f9-bc1b-d2b6446ddfaf","Type":"ContainerStarted","Data":"6ab507648ce12be648009ee934ef0a972fa7fc27d1b0d574ba300843033cbd77"}
Oct 30 19:25:31.528834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.528800    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" event={"ID":"61456446-5707-4a7d-8f0d-bc34fe251d44","Type":"ContainerStarted","Data":"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"}
Oct 30 19:25:31.528937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.528859    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" event={"ID":"61456446-5707-4a7d-8f0d-bc34fe251d44","Type":"ContainerStarted","Data":"460842d646562bfdff98cde4313da0a9d1725a4971b1422b23197696ee5c37be"}
Oct 30 19:25:31.568838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.568787    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podStartSLOduration=1.56877034 podStartE2EDuration="1.56877034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:31.566704961 +0000 UTC m=+1279.602217426" watchObservedRunningTime="2026-10-30 19:25:31.56877034 +0000 UTC m=+1279.604282808"
Oct 30 19:25:31.610049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.609989    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podStartSLOduration=1.609973616 podStartE2EDuration="1.609973616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:31.6088114 +0000 UTC m=+1279.644323864" watchObservedRunningTime="2026-10-30 19:25:31.609973616 +0000 UTC m=+1279.645486079"
Oct 30 19:25:31.662732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.662660    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:31.830657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:31.830557    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:32.662924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:32.662893    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:32.663318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:32.662944    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:32.831318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:32.831284    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:32.831475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:32.831338    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:33.661017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:33.660982    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:33.661195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:33.661036    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:33.830547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:33.830201    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:33.830547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:33.830250    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:34.661199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:34.661152    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:34.661394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:34.661212    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:34.830221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:34.830190    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:34.830443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:34.830421    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:35.661537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:35.661509    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:35.662071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:35.661697    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:35.830695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:35.830660    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:35.830873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:35.830711    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:36.660697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:36.660661    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:36.660893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:36.660728    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:36.837621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:36.831013    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:36.837621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:36.831075    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:37.660645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:37.660588    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:25:37.660975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:37.660660    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:25:37.830003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:37.829965    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:25:37.830193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:37.830032    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:25:38.664666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.664638    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:38.665149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.665015    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:38.665639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.665616    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:25:38.835473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.835444    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:38.835838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.835815    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:38.836562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:38.836533    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:25:40.926523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:40.926488    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:25:43.899621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:43.898675    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:43.960759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:43.960683    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:44.016553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:44.015682    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:45.163746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.163714    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"]
Oct 30 19:25:45.168023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.168004    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.187072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.187050    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"]
Oct 30 19:25:45.194315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.194296    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:45.194455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.194407    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:25:45.252230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.252206    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.252343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.252236    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.252448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.252422    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kf98\" (UniqueName: \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.353276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.353253    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kf98\" (UniqueName: \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.353418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.353390    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.353418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.353414    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.354065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.354045    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.355717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.355699    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.371920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.371898    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kf98\" (UniqueName: \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\") pod \"maas-api-revoke-sub-4f138771-fnbd6\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.482374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.482319    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:45.784593 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:45.784568    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"]
Oct 30 19:25:45.785713 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:25:45.785684    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fbc46f_1d6f_4894_8158_c58270cc0b5a.slice/crio-c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af WatchSource:0}: Error finding container c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af: Status 404 returned error can't find the container with id c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af
Oct 30 19:25:46.709589 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:46.709556    4320 generic.go:356] "Generic (PLEG): container finished" podID="02fbc46f-1d6f-4894-8158-c58270cc0b5a" containerID="251f32663af51a9068aee0b2ce48fedba7d675e9cbb445fdeddd362e8499e93a" exitCode=0
Oct 30 19:25:46.710048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:46.709663    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6" event={"ID":"02fbc46f-1d6f-4894-8158-c58270cc0b5a","Type":"ContainerDied","Data":"251f32663af51a9068aee0b2ce48fedba7d675e9cbb445fdeddd362e8499e93a"}
Oct 30 19:25:46.710048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:46.709717    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6" event={"ID":"02fbc46f-1d6f-4894-8158-c58270cc0b5a","Type":"ContainerStarted","Data":"c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af"}
Oct 30 19:25:47.875238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.875199    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:47.982754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.982673    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\") pod \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") "
Oct 30 19:25:47.982916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.982767    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\" (UniqueName: \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\") pod \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") "
Oct 30 19:25:47.982916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.982826    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\") pod \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\" (UID: \"02fbc46f-1d6f-4894-8158-c58270cc0b5a\") "
Oct 30 19:25:47.983189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.983164    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca" pod "02fbc46f-1d6f-4894-8158-c58270cc0b5a" (UID: "02fbc46f-1d6f-4894-8158-c58270cc0b5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:47.983322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.983299    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:25:47.984787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.984764    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca" pod "02fbc46f-1d6f-4894-8158-c58270cc0b5a" (UID: "02fbc46f-1d6f-4894-8158-c58270cc0b5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:47.984957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:47.984938    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98" pod "02fbc46f-1d6f-4894-8158-c58270cc0b5a" (UID: "02fbc46f-1d6f-4894-8158-c58270cc0b5a"). InnerVolumeSpecName "kube-api-access-5kf98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:48.084203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:48.084176    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02fbc46f-1d6f-4894-8158-c58270cc0b5a-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:25:48.084203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:48.084201    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kf98\" (UniqueName: \"kubernetes.io/projected/02fbc46f-1d6f-4894-8158-c58270cc0b5a-kube-api-access-5kf98\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:25:48.723783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:48.723747    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6" event={"ID":"02fbc46f-1d6f-4894-8158-c58270cc0b5a","Type":"ContainerDied","Data":"c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af"}
Oct 30 19:25:48.723783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:48.723768    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"
Oct 30 19:25:48.723998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:25:48.723797    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d1eb6e2aa04d978eed8bde212d0e7a56dff8890246dc41e2dc6a5e918899af"
Oct 30 19:25:58.646819 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:25:58.646731    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:26:14.849811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.849778    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"]
Oct 30 19:26:14.850391 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.850376    4320 cpu_manager.go:414] "containerMap: removing container" podUID="02fbc46f-1d6f-4894-8158-c58270cc0b5a" containerName="revoke-keys"
Oct 30 19:26:14.850435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.850394    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fbc46f-1d6f-4894-8158-c58270cc0b5a" containerName="revoke-keys"
Oct 30 19:26:14.850500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.850492    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fbc46f-1d6f-4894-8158-c58270cc0b5a" containerName="revoke-keys"
Oct 30 19:26:14.854359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.854335    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:14.871017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.870985    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"]
Oct 30 19:26:14.878871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.878820    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:14.879003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.878886    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:26:14.959995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.959966    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:14.960124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.960019    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqvs\" (UniqueName: \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:14.960171 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:14.960121    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.061728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.061701    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.061880 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.061761    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqvs\" (UniqueName: \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.061880 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.061799    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.062432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.062407    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.064545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.064523    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.080424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.080403    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqvs\" (UniqueName: \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\") pod \"maas-api-revoke-sub-049f94ce-pchbl\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.168699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.168673    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:15.462101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.462031    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:26:15.467890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.467868    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.474006 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.473983    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"]
Oct 30 19:26:15.478190 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:26:15.478161    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6719c0f4_adfc_4b10_85d7_b9b9d224d9b7.slice/crio-2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c WatchSource:0}: Error finding container 2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c: Status 404 returned error can't find the container with id 2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c
Oct 30 19:26:15.492762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.492728    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:26:15.567760 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.567731    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.567867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.567765    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kjd\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.567867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.567786    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.567943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.567901    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.567993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.567979    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.568057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.568045    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.568093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.568075    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.568129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.568094    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.568206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.568194    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669384    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669441    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669469    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669486    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669578    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669666    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669698    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kjd\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669725    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669772    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669850    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.669964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.669897    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.670300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.670201    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.670518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.670489    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.670818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.670799    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.672582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.672554    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.674145 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.674123    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.691837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.691818    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.692286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.692265    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kjd\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.799968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.799948    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:15.906372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.906334    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl" event={"ID":"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7","Type":"ContainerStarted","Data":"d5dfc4e4f2b508f761cf4eb217b298995d36be922c0556d1a681ebf7ce67301e"}
Oct 30 19:26:15.906746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.906399    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl" event={"ID":"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7","Type":"ContainerStarted","Data":"2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c"}
Oct 30 19:26:15.948287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:15.948237    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl" podStartSLOduration=1.9482220049999999 podStartE2EDuration="1.948222005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:15.947012858 +0000 UTC m=+1323.982525321" watchObservedRunningTime="2026-10-30 19:26:15.948222005 +0000 UTC m=+1323.983734532"
Oct 30 19:26:16.119298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.119272    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:26:16.120954 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:26:16.120929    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862f1849_b8bd_4345_9289_e63b1d26d5fe.slice/crio-9f0741d02413d07170eec3521b7fa1c0e732802a23422f609e53e3f77bb50de8 WatchSource:0}: Error finding container 9f0741d02413d07170eec3521b7fa1c0e732802a23422f609e53e3f77bb50de8: Status 404 returned error can't find the container with id 9f0741d02413d07170eec3521b7fa1c0e732802a23422f609e53e3f77bb50de8
Oct 30 19:26:16.123201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.123166    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:16.123288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.123230    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:16.123288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.123258    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:16.913923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.913886    4320 generic.go:356] "Generic (PLEG): container finished" podID="6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" containerID="d5dfc4e4f2b508f761cf4eb217b298995d36be922c0556d1a681ebf7ce67301e" exitCode=0
Oct 30 19:26:16.914375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.914012    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl" event={"ID":"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7","Type":"ContainerDied","Data":"d5dfc4e4f2b508f761cf4eb217b298995d36be922c0556d1a681ebf7ce67301e"}
Oct 30 19:26:16.915809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.915784    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" event={"ID":"862f1849-b8bd-4345-9289-e63b1d26d5fe","Type":"ContainerStarted","Data":"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"}
Oct 30 19:26:16.915906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.915824    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" event={"ID":"862f1849-b8bd-4345-9289-e63b1d26d5fe","Type":"ContainerStarted","Data":"9f0741d02413d07170eec3521b7fa1c0e732802a23422f609e53e3f77bb50de8"}
Oct 30 19:26:16.997397 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:16.997336    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podStartSLOduration=1.9973187110000001 podStartE2EDuration="1.997318711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:16.996568558 +0000 UTC m=+1325.032081023" watchObservedRunningTime="2026-10-30 19:26:16.997318711 +0000 UTC m=+1325.032831176"
Oct 30 19:26:17.801114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:17.801082    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:17.802516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:17.802494    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:17.802580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:17.802541    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:18.125022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.125000    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:18.214966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.214923    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\" (UniqueName: \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\") pod \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") "
Oct 30 19:26:18.215805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.215765    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\") pod \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") "
Oct 30 19:26:18.216307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.216285    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\") pod \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\" (UID: \"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7\") "
Oct 30 19:26:18.217711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.217687    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca" pod "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" (UID: "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:18.220531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.220505    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs" pod "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" (UID: "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7"). InnerVolumeSpecName "kube-api-access-7vqvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:18.227022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.226999    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca" pod "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" (UID: "6719c0f4-adfc-4b10-85d7-b9b9d224d9b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:18.318616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.318576    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:18.318802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.318627    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqvs\" (UniqueName: \"kubernetes.io/projected/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-kube-api-access-7vqvs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:18.318802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.318639    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:18.800587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.800557    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:18.800761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.800627    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:18.929503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.929476    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"
Oct 30 19:26:18.929503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.929479    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl" event={"ID":"6719c0f4-adfc-4b10-85d7-b9b9d224d9b7","Type":"ContainerDied","Data":"2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c"}
Oct 30 19:26:18.929779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:18.929532    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7664d77ad1fe5cc91ae4bd9eefeaf578d4038715dd78bf71f6e94f42dc458c"
Oct 30 19:26:19.801279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:19.801240    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:19.801809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:19.801296    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:20.800795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:20.800758    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:20.800978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:20.800813    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:21.800417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:21.800363    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:21.800964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:21.800441    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:22.800806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:22.800778    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:26:22.801178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:22.800831    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:26:22.993755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:22.993715    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:26:23.804726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:23.804693    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:23.805156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:23.805137    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:23.805940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:23.805923    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:26:25.585386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.574740    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:26:25.585386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.575713    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" containerName="revoke-keys"
Oct 30 19:26:25.585386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.575734    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" containerName="revoke-keys"
Oct 30 19:26:25.585386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.575929    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" containerName="revoke-keys"
Oct 30 19:26:25.585386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.580276    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.634773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.633746    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:26:25.686803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.686773    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.729064    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.729221    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752322    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752430    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752461    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752504    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752634    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w79w\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752664    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752737    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752813    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.881727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.752841    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897000    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897046    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897086    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897195    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897220    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897267    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897337    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897364    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897391    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897417    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85sr\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897476    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897561    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w79w\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897591    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897716    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897837    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897867    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.897894    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.898830    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.901506    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.901870    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.902369    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:25.921476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:25.915199    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004665    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004715    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004750    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004778    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85sr\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004829    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.004969    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.005064    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.005125    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.005162    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.005768    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.006437    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.006709    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.009124    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.014164    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.052570    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.059137    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.059218    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.068175    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85sr\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.091461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.091390    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.152224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.152001    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.153002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.152951    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.163663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.158203    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w79w\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.258164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.232518    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.333621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.316830    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:26.337616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.334826    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:26.363618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.347676    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:26.840708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.840562    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:26:26.844097 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:26:26.844070    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9580643d_b813_497c_b82c_a428837ec65e.slice/crio-f575e163bc91432cdad3cd5a555cee2aa6815308115b9b7b1bc1d0d52af70098 WatchSource:0}: Error finding container f575e163bc91432cdad3cd5a555cee2aa6815308115b9b7b1bc1d0d52af70098: Status 404 returned error can't find the container with id f575e163bc91432cdad3cd5a555cee2aa6815308115b9b7b1bc1d0d52af70098
Oct 30 19:26:26.846561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.846530    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:26.846679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.846593    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:26.846679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.846636    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:26.945334 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:26:26.945302    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f2b7b8_ec87_4055_8257_7074925e1b63.slice/crio-1c7ebfcdeee552af0899756f21599f6e17b1fee69fd43481de07a1ff0cbaa9e0 WatchSource:0}: Error finding container 1c7ebfcdeee552af0899756f21599f6e17b1fee69fd43481de07a1ff0cbaa9e0: Status 404 returned error can't find the container with id 1c7ebfcdeee552af0899756f21599f6e17b1fee69fd43481de07a1ff0cbaa9e0
Oct 30 19:26:26.948359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.947842    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:26:26.952740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.952034    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:26.952740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.952112    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:26.952740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:26.952153    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:26:27.175923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.175876    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" event={"ID":"59f2b7b8-ec87-4055-8257-7074925e1b63","Type":"ContainerStarted","Data":"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"}
Oct 30 19:26:27.176115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.175951    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" event={"ID":"59f2b7b8-ec87-4055-8257-7074925e1b63","Type":"ContainerStarted","Data":"1c7ebfcdeee552af0899756f21599f6e17b1fee69fd43481de07a1ff0cbaa9e0"}
Oct 30 19:26:27.177810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.177778    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" event={"ID":"9580643d-b813-497c-b82c-a428837ec65e","Type":"ContainerStarted","Data":"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"}
Oct 30 19:26:27.177915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.177833    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" event={"ID":"9580643d-b813-497c-b82c-a428837ec65e","Type":"ContainerStarted","Data":"f575e163bc91432cdad3cd5a555cee2aa6815308115b9b7b1bc1d0d52af70098"}
Oct 30 19:26:27.221035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.220934    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podStartSLOduration=2.220915019 podStartE2EDuration="2.220915019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:27.215985229 +0000 UTC m=+1335.251497694" watchObservedRunningTime="2026-10-30 19:26:27.220915019 +0000 UTC m=+1335.256427484"
Oct 30 19:26:27.259698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.259632    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podStartSLOduration=2.259618216 podStartE2EDuration="2.259618216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:27.257446924 +0000 UTC m=+1335.292959388" watchObservedRunningTime="2026-10-30 19:26:27.259618216 +0000 UTC m=+1335.295130677"
Oct 30 19:26:27.318771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.318731    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:27.335287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.335260    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:27.336850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.336813    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:27.336978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:27.336879    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:28.322888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:28.322720    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:28.322888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:28.322780    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:28.335288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:28.335260    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:28.335402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:28.335307    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:28.650367 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:26:28.650317    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:26:29.318621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:29.318081    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:29.318621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:29.318156    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:29.335855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:29.335823    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:29.336181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:29.335873    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:30.318534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:30.318005    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:30.318534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:30.318074    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:30.335735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:30.335707    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:30.335856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:30.335745    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:31.317989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:31.317949    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:31.318614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:31.318560    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:31.335335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:31.335309    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:31.335425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:31.335345    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:32.319369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:32.318918    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:32.319369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:32.318977    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:32.336214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:32.336191    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:32.336392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:32.336374    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:33.317929 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:33.317890    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:26:33.318109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:33.317951    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:26:33.335381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:33.335348    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:26:33.335853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:33.335406    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 19:26:34.628145 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.628109    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:34.628976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.628940    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:34.630282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.630248    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:26:34.738887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.738851    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:34.739754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.739729    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:34.745230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:34.745178    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:26:45.256621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.255566    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:26:45.347618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.343682    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:26:45.347618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.344705    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:45.371139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.370382    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:45.480632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.479721    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:45.902706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.900427    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:45.975659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:45.974651    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:26:46.047751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:46.046688    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:46.371447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:46.371346    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:46.379639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:46.372580    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:26:46.482834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:46.480631    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:46.482834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:46.481341    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:26:47.110689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.054282    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:26:47.110689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.055422    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:26:47.177773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.177722    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.491352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.281227    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.491971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.295924    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.492768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.351335    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.494051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.376735    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:26:47.494761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.486932    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.602708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.602676    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:26:47.742022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.741941    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:26:47.742686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:47.742587    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:26:49.376418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.376387    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"]
Oct 30 19:26:49.381091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.381072    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.402550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.402522    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:49.405001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.404975    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"]
Oct 30 19:26:49.411752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.411733    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:26:49.446480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.446456    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.446635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.446488    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6jx\" (UniqueName: \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.446635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.446623    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.547942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.547919    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.548065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.547951    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6jx\" (UniqueName: \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.548065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.548008    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.548713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.548690    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.550281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.550255    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.569425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.569400    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6jx\" (UniqueName: \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\") pod \"maas-api-revoke-sub-cc73524f-8j5xx\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:49.700382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:49.700349    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:50.006687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:50.004830    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"]
Oct 30 19:26:50.019449 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:26:50.019418    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa16902_d511_43d9_8ca6_665d101f87ad.slice/crio-1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f WatchSource:0}: Error finding container 1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f: Status 404 returned error can't find the container with id 1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f
Oct 30 19:26:50.860014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:50.859982    4320 generic.go:356] "Generic (PLEG): container finished" podID="0aa16902-d511-43d9-8ca6-665d101f87ad" containerID="056cb69aa464c880dd96984f4a0a13e9f2ea8c5b3f829d7024cc9bd3fdc5f1ce" exitCode=0
Oct 30 19:26:50.860349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:50.860020    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx" event={"ID":"0aa16902-d511-43d9-8ca6-665d101f87ad","Type":"ContainerDied","Data":"056cb69aa464c880dd96984f4a0a13e9f2ea8c5b3f829d7024cc9bd3fdc5f1ce"}
Oct 30 19:26:50.860349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:50.860069    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx" event={"ID":"0aa16902-d511-43d9-8ca6-665d101f87ad","Type":"ContainerStarted","Data":"1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f"}
Oct 30 19:26:52.040014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.039991    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:52.086342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.086322    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\") pod \"0aa16902-d511-43d9-8ca6-665d101f87ad\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") "
Oct 30 19:26:52.086456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.086370    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\") pod \"0aa16902-d511-43d9-8ca6-665d101f87ad\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") "
Oct 30 19:26:52.086528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.086461    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\" (UniqueName: \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\") pod \"0aa16902-d511-43d9-8ca6-665d101f87ad\" (UID: \"0aa16902-d511-43d9-8ca6-665d101f87ad\") "
Oct 30 19:26:52.086748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.086724    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca" pod "0aa16902-d511-43d9-8ca6-665d101f87ad" (UID: "0aa16902-d511-43d9-8ca6-665d101f87ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:52.087030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.087009    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:52.088612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.088564    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx" pod "0aa16902-d511-43d9-8ca6-665d101f87ad" (UID: "0aa16902-d511-43d9-8ca6-665d101f87ad"). InnerVolumeSpecName "kube-api-access-tw6jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:52.088718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.088699    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca" pod "0aa16902-d511-43d9-8ca6-665d101f87ad" (UID: "0aa16902-d511-43d9-8ca6-665d101f87ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:52.188440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.188416    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6jx\" (UniqueName: \"kubernetes.io/projected/0aa16902-d511-43d9-8ca6-665d101f87ad-kube-api-access-tw6jx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:52.188440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.188438    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa16902-d511-43d9-8ca6-665d101f87ad-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:26:52.874086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.874049    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx" event={"ID":"0aa16902-d511-43d9-8ca6-665d101f87ad","Type":"ContainerDied","Data":"1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f"}
Oct 30 19:26:52.874086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.874069    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"
Oct 30 19:26:52.874279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:26:52.874123    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fed0a99dabc04334f88f1755a0ab7f6e37bcd591f2cd5a38f6c2f8b89b0542f"
Oct 30 19:26:58.645086 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:26:58.645051    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:27:04.082155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:04.082116    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:27:06.393623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.388830    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.501626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.500677    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.609620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.608716    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.898370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.898332    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:27:06.941714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.941670    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:06.960518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.960485    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:27:06.990910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:06.990860    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.002463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.001730    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.206890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.206393    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.254784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.253679    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.394644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.392679    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:07.394644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.393430    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:07.509806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.509706    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:27:07.515716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.515690    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:07.602051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.602017    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:27:07.808187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.796892    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:07.812295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:07.812119    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:17.731070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.731034    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"]
Oct 30 19:27:17.731926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.731907    4320 cpu_manager.go:414] "containerMap: removing container" podUID="0aa16902-d511-43d9-8ca6-665d101f87ad" containerName="revoke-keys"
Oct 30 19:27:17.732003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.731928    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa16902-d511-43d9-8ca6-665d101f87ad" containerName="revoke-keys"
Oct 30 19:27:17.732119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.732105    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa16902-d511-43d9-8ca6-665d101f87ad" containerName="revoke-keys"
Oct 30 19:27:17.736051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.736033    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.753178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.753154    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"]
Oct 30 19:27:17.759784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.759762    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:17.760691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.760669    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:27:17.807737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.807702    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqxs\" (UniqueName: \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.807880 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.807769    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.807941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.807880    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.909134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.909097    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqxs\" (UniqueName: \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.909250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.909164    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.909250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.909231    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.909829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.909800    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.911511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.911489    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:17.933962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:17.933940    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqxs\" (UniqueName: \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\") pod \"maas-api-revoke-sub-8fecc27e-bkkcf\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:18.067062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:18.066973    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:18.372336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:18.372308    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"]
Oct 30 19:27:18.373369 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:27:18.373339    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b9e04e_17ca_45a8_8847_de06cd479747.slice/crio-9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9 WatchSource:0}: Error finding container 9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9: Status 404 returned error can't find the container with id 9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9
Oct 30 19:27:19.088339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:19.088179    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf" event={"ID":"90b9e04e-17ca-45a8-8847-de06cd479747","Type":"ContainerStarted","Data":"85c55a0f0a28a04794f20a5dc632a4991a25c6f2a7ec3cf9e6d3736d048d36dd"}
Oct 30 19:27:19.088339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:19.088251    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf" event={"ID":"90b9e04e-17ca-45a8-8847-de06cd479747","Type":"ContainerStarted","Data":"9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9"}
Oct 30 19:27:19.128305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:19.128251    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf" podStartSLOduration=2.128235965 podStartE2EDuration="2.128235965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:19.125992197 +0000 UTC m=+1387.161504661" watchObservedRunningTime="2026-10-30 19:27:19.128235965 +0000 UTC m=+1387.163748429"
Oct 30 19:27:20.097004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:20.096968    4320 generic.go:356] "Generic (PLEG): container finished" podID="90b9e04e-17ca-45a8-8847-de06cd479747" containerID="85c55a0f0a28a04794f20a5dc632a4991a25c6f2a7ec3cf9e6d3736d048d36dd" exitCode=0
Oct 30 19:27:20.097395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:20.097278    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf" event={"ID":"90b9e04e-17ca-45a8-8847-de06cd479747","Type":"ContainerDied","Data":"85c55a0f0a28a04794f20a5dc632a4991a25c6f2a7ec3cf9e6d3736d048d36dd"}
Oct 30 19:27:21.249648 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.249624    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:21.350636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.350562    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\" (UniqueName: \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\") pod \"90b9e04e-17ca-45a8-8847-de06cd479747\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") "
Oct 30 19:27:21.350780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.350745    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\") pod \"90b9e04e-17ca-45a8-8847-de06cd479747\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") "
Oct 30 19:27:21.350780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.350766    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\") pod \"90b9e04e-17ca-45a8-8847-de06cd479747\" (UID: \"90b9e04e-17ca-45a8-8847-de06cd479747\") "
Oct 30 19:27:21.351108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.351076    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca" pod "90b9e04e-17ca-45a8-8847-de06cd479747" (UID: "90b9e04e-17ca-45a8-8847-de06cd479747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:21.351257 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.351242    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:21.352698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.352671    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs" pod "90b9e04e-17ca-45a8-8847-de06cd479747" (UID: "90b9e04e-17ca-45a8-8847-de06cd479747"). InnerVolumeSpecName "kube-api-access-9hqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:21.352845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.352820    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca" pod "90b9e04e-17ca-45a8-8847-de06cd479747" (UID: "90b9e04e-17ca-45a8-8847-de06cd479747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:21.452921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.452894    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hqxs\" (UniqueName: \"kubernetes.io/projected/90b9e04e-17ca-45a8-8847-de06cd479747-kube-api-access-9hqxs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:21.452921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:21.452917    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90b9e04e-17ca-45a8-8847-de06cd479747-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:22.111314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:22.111289    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"
Oct 30 19:27:22.111453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:22.111292    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf" event={"ID":"90b9e04e-17ca-45a8-8847-de06cd479747","Type":"ContainerDied","Data":"9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9"}
Oct 30 19:27:22.111453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:22.111410    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6c5dac4e4c1d1c43859b7bd27ef4e64f0d6aaf9d8dc0d0d94b479f5b569dc9"
Oct 30 19:27:27.606697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.606568    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:27.785260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.785190    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:27.819035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.812202    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:27.831914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.831859    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:27.892121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.890942    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:27:27.946495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.946456    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:27:27.959710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:27.959670    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.001714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.001672    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.144791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.144684    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:27:28.253622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.252704    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.498900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.497751    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.610746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.610693    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.612058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.612033    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:27:28.875676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.812702    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.875676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.813446    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:28.875676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.848926    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:27:28.875676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.849309    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:28.981726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.903725    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.981726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.949728    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:27:28.981726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.978747    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.981726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.979367    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:27:28.994656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.982277    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:28.994656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:28.982626    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:27:29.037636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.002799    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:29.037636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.010353    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:27:29.202622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.201713    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Oct 30 19:27:29.295642 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:27:29.288777    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:27:29.491973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.491880    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:27:29.498234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.497757    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:27:29.501949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.501402    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:27:29.502283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.502265    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:27:29.559633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.555722    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:29.658622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.556333    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:27:29.821286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:29.783140    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.011621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.002055    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 19:27:30.024982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.024935    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.031319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.031284    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:27:30.156420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.046155    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:27:30.156420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.047146    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.156420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.065479    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.156420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.087482    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.532680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.526284    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.532680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.526625    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:27:30.564822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.564043    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.564822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.564126    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:27:30.788228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.788083    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:30.973621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:30.947285    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:27:31.024066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.017909    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:27:31.024066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.018863    4320 cpu_manager.go:414] "containerMap: removing container" podUID="90b9e04e-17ca-45a8-8847-de06cd479747" containerName="revoke-keys"
Oct 30 19:27:31.024066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.018885    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b9e04e-17ca-45a8-8847-de06cd479747" containerName="revoke-keys"
Oct 30 19:27:31.024066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.019066    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b9e04e-17ca-45a8-8847-de06cd479747" containerName="revoke-keys"
Oct 30 19:27:31.034800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.029570    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:27:31.034800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.030768    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:31.053706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.047717    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.181618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.151780    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:27:31.181618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.176122    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:27:31.273640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.273588    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:27:31.274165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.274140    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:27:31.279318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.279293    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:31.338667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.335245    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:31.353149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353076    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353146    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353196    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353266    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353299    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353367    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353451    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353520    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.353659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.353564    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc6s\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.362911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.362885    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:27:31.411036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.411004    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:27:31.424129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.424100    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:27:31.425073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.425053    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:27:31.452528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.451844    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:27:31.457100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457075    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457248 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457233    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457376    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457525    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457629    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457779    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.457969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.457956    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.458126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.458114    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.458214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.458201    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc6s\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.459444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.459423    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.460205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.460186    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.462252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.462069    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.462647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.462622    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.465481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.464926    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.469023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.468982    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.473546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.473512    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.475912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.475829    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:27:31.486391 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.486350    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:27:31.508570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.508467    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc6s\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.512398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.512349    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.517910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.517848    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:31.549687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.549664    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:31.640206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.640181    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:31.968820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.968792    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:27:31.969640 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:27:31.969610    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a61ddab_2049_47ca_8fe6_4ffaabb1251a.slice/crio-6b93c64286b8fdbab6faab0a33b9408a414d26dec354f970f6ec91e36bd162e5 WatchSource:0}: Error finding container 6b93c64286b8fdbab6faab0a33b9408a414d26dec354f970f6ec91e36bd162e5: Status 404 returned error can't find the container with id 6b93c64286b8fdbab6faab0a33b9408a414d26dec354f970f6ec91e36bd162e5
Oct 30 19:27:31.972104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.972069    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:27:31.972190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.972136    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:27:31.972190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:31.972166    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:27:32.260141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:32.260039    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" event={"ID":"7a61ddab-2049-47ca-8fe6-4ffaabb1251a","Type":"ContainerStarted","Data":"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"}
Oct 30 19:27:32.260141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:32.260113    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" event={"ID":"7a61ddab-2049-47ca-8fe6-4ffaabb1251a","Type":"ContainerStarted","Data":"6b93c64286b8fdbab6faab0a33b9408a414d26dec354f970f6ec91e36bd162e5"}
Oct 30 19:27:32.307924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:32.307858    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podStartSLOduration=2.30783854 podStartE2EDuration="2.30783854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:32.304869828 +0000 UTC m=+1400.340382293" watchObservedRunningTime="2026-10-30 19:27:32.30783854 +0000 UTC m=+1400.343351067"
Oct 30 19:27:32.647752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:32.647702    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:33.642320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:33.642284    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:33.642732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:33.642350    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:34.641417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:34.641383    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:34.641583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:34.641445    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:35.640642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:35.640591    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:35.641055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:35.640675    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:36.640789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:36.640757    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:36.641193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:36.640817    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:37.394175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.394139    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"]
Oct 30 19:27:37.399451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.399425    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"]
Oct 30 19:27:37.399628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.399518    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.425500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.425475    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:37.425500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.425507    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:27:37.461966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.461930    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.462088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.462009    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.462169 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.462154    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t44k\" (UniqueName: \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.563501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.563471    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t44k\" (UniqueName: \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.563700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.563568    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.563700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.563625    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.564436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.564407    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.566549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.566527    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.599586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.599559    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t44k\" (UniqueName: \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\") pod \"maas-api-revoke-sub-fc23c18d-g6q6m\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:37.641255 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.641228    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:37.641566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.641282    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:37.719251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:37.719189    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:38.041800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:38.041766    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"]
Oct 30 19:27:38.049451 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:27:38.049426    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7602b3b_7fe2_4aaf_819a_a8583f52ffa6.slice/crio-7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed WatchSource:0}: Error finding container 7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed: Status 404 returned error can't find the container with id 7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed
Oct 30 19:27:38.340130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:38.340067    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m" event={"ID":"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6","Type":"ContainerStarted","Data":"7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed"}
Oct 30 19:27:38.641576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:38.641544    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:27:38.642025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:38.641593    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 19:27:39.350673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:39.350544    4320 generic.go:356] "Generic (PLEG): container finished" podID="f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" containerID="768d70694e14482ab6b27269c60246cae016274de03ca8b5a5fef4ff61607d49" exitCode=0
Oct 30 19:27:39.350673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:39.350642    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m" event={"ID":"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6","Type":"ContainerDied","Data":"768d70694e14482ab6b27269c60246cae016274de03ca8b5a5fef4ff61607d49"}
Oct 30 19:27:39.644553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:39.644525    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:39.644957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:39.644833    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:39.645664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:39.645646    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:27:40.519853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.519826    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:40.610572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.610507    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\") pod \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") "
Oct 30 19:27:40.610758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.610744    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\") pod \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") "
Oct 30 19:27:40.610801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.610794    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\" (UniqueName: \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\") pod \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\" (UID: \"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6\") "
Oct 30 19:27:40.610908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.610871    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca" pod "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" (UID: "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:40.611229 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.611201    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:40.612836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.612804    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca" pod "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" (UID: "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:40.612928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.612909    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k" pod "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" (UID: "f7602b3b-7fe2-4aaf-819a-a8583f52ffa6"). InnerVolumeSpecName "kube-api-access-7t44k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:40.712273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.712239    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:40.712273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:40.712267    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t44k\" (UniqueName: \"kubernetes.io/projected/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6-kube-api-access-7t44k\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:41.366667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:41.366624    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m" event={"ID":"f7602b3b-7fe2-4aaf-819a-a8583f52ffa6","Type":"ContainerDied","Data":"7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed"}
Oct 30 19:27:41.366667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:41.366668    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"
Oct 30 19:27:41.366859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:41.366717    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c31fbfbad70c22de9ab24e25a9427c7c6e7b091c8189bb8a45b4fb482a217ed"
Oct 30 19:27:43.779018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.778982    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"]
Oct 30 19:27:43.781486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.779728    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" containerName="revoke-keys"
Oct 30 19:27:43.781486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.779744    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" containerName="revoke-keys"
Oct 30 19:27:43.781486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.779872    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" containerName="revoke-keys"
Oct 30 19:27:43.783055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.783036    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.801923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.801898    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"]
Oct 30 19:27:43.809000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.808980    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:27:43.809107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.809029    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:43.851245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.851217    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqplw\" (UniqueName: \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.851457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.851435    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.851543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.851492    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.952948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.952921    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.953062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.952965    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.953062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.953041    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqplw\" (UniqueName: \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.953677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.953657    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.955277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.955256    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:43.971935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:43.971912    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqplw\" (UniqueName: \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\") pod \"maas-api-revoke-sub-9049d375-6grgn\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:44.096710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:44.096651    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:44.412254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:44.412220    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"]
Oct 30 19:27:44.423543 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:27:44.423514    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7883575b_ab2d_4b62_a330_7e9c018078df.slice/crio-53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649 WatchSource:0}: Error finding container 53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649: Status 404 returned error can't find the container with id 53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649
Oct 30 19:27:45.394895 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:45.394859    4320 generic.go:356] "Generic (PLEG): container finished" podID="7883575b-ab2d-4b62-a330-7e9c018078df" containerID="bf8f7f91e4744c32edf1fcee91389db5217938c1352eb2ba96611636eb909545" exitCode=0
Oct 30 19:27:45.395214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:45.394939    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn" event={"ID":"7883575b-ab2d-4b62-a330-7e9c018078df","Type":"ContainerDied","Data":"bf8f7f91e4744c32edf1fcee91389db5217938c1352eb2ba96611636eb909545"}
Oct 30 19:27:45.395214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:45.395026    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn" event={"ID":"7883575b-ab2d-4b62-a330-7e9c018078df","Type":"ContainerStarted","Data":"53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649"}
Oct 30 19:27:46.575461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.575435    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:46.692720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.692650    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\") pod \"7883575b-ab2d-4b62-a330-7e9c018078df\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") "
Oct 30 19:27:46.692868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.692757    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\" (UniqueName: \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\") pod \"7883575b-ab2d-4b62-a330-7e9c018078df\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") "
Oct 30 19:27:46.692868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.692802    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\") pod \"7883575b-ab2d-4b62-a330-7e9c018078df\" (UID: \"7883575b-ab2d-4b62-a330-7e9c018078df\") "
Oct 30 19:27:46.693004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.692981    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca" pod "7883575b-ab2d-4b62-a330-7e9c018078df" (UID: "7883575b-ab2d-4b62-a330-7e9c018078df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:46.693489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.693467    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:46.694889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.694868    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw" pod "7883575b-ab2d-4b62-a330-7e9c018078df" (UID: "7883575b-ab2d-4b62-a330-7e9c018078df"). InnerVolumeSpecName "kube-api-access-cqplw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:46.694971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.694914    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca" pod "7883575b-ab2d-4b62-a330-7e9c018078df" (UID: "7883575b-ab2d-4b62-a330-7e9c018078df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:46.770023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.769995    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:27:46.794817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.794784    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqplw\" (UniqueName: \"kubernetes.io/projected/7883575b-ab2d-4b62-a330-7e9c018078df-kube-api-access-cqplw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:46.794817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:46.794813    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7883575b-ab2d-4b62-a330-7e9c018078df-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:27:47.410225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:47.410196    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"
Oct 30 19:27:47.410400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:47.410196    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn" event={"ID":"7883575b-ab2d-4b62-a330-7e9c018078df","Type":"ContainerDied","Data":"53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649"}
Oct 30 19:27:47.410400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:47.410327    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53988c92fbb2a3ee127c1e10bdda73f7506e6627c173e00e299f8f2316596649"
Oct 30 19:27:49.055739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.055696    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:49.125623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.123726    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:49.140919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.140860    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:27:49.166769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.166381    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:27:49.182953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.182580    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:49.281788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.281737    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:49.283528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.281674    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:27:49.532848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.532765    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:49.572891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.572833    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:27:49.594778 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:49.594728    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:50.056734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.056683    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:50.058522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.058502    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:27:50.329994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.304920    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:50.329994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.318848    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:27:50.430505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.430458    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:50.432032 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.432005    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:27:50.626858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.626737    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:50.646230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.646178    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:27:50.684293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.684264    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:50.786089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.786026    4320 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:27:50.786409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:50.786378    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:51.069741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.069692    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:27:51.305692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.305640    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:51.614661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.594683    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:51.650794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.650751    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:51.696640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.692788    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:51.776016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:51.775969    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.087280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.087236    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.137307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.137263    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.318962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.317716    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:27:52.341102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.340684    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:27:52.357794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.354530    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.595766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.594919    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.660713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.657933    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:27:52.660713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.658652    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:27:52.694791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.694753    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.801288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.801259    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:27:52.806209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.801952    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:52.808969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.807374    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:27:52.832222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:52.832185    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:27:53.084630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.081661    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:27:53.114879 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.114839    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:27:53.158635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.155443    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:53.158635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.155995    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:53.195960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.195931    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:27:53.243393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.243247    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:27:53.259503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:27:53.259477    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:27:58.644433 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:27:58.644398    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:28:00.795123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.795091    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"]
Oct 30 19:28:00.795700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.795684    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7883575b-ab2d-4b62-a330-7e9c018078df" containerName="revoke-keys"
Oct 30 19:28:00.795700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.795700    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7883575b-ab2d-4b62-a330-7e9c018078df" containerName="revoke-keys"
Oct 30 19:28:00.795808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.795799    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7883575b-ab2d-4b62-a330-7e9c018078df" containerName="revoke-keys"
Oct 30 19:28:00.800402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.800383    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.821839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.821813    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"]
Oct 30 19:28:00.825652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.825625    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:00.826043 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.825803    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:28:00.859692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.859666    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.859873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.859855    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jz2\" (UniqueName: \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.859950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.859884    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.960916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.960887    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jz2\" (UniqueName: \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.961020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.960926    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.961141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.961119    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.961541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.961517    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.963314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.963295    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:00.980546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:00.980526    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jz2\" (UniqueName: \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\") pod \"maas-api-revoke-sub-2eb10009-kp7zb\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:01.126932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:01.126842    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:01.442538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:01.442511    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"]
Oct 30 19:28:01.443643 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:01.443614    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729bea98_1e30_4252_96d6_b484a139050d.slice/crio-3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be WatchSource:0}: Error finding container 3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be: Status 404 returned error can't find the container with id 3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be
Oct 30 19:28:01.887918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:01.887881    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb" event={"ID":"729bea98-1e30-4252-96d6-b484a139050d","Type":"ContainerStarted","Data":"cc5f5d7c8c4fd3e6f2dde1b453c779ba4617e8b7b623afe9dbd8095defa45737"}
Oct 30 19:28:01.888264 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:01.887937    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb" event={"ID":"729bea98-1e30-4252-96d6-b484a139050d","Type":"ContainerStarted","Data":"3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be"}
Oct 30 19:28:01.929040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:01.928990    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb" podStartSLOduration=1.928976931 podStartE2EDuration="1.928976931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:01.925780909 +0000 UTC m=+1429.961293376" watchObservedRunningTime="2026-10-30 19:28:01.928976931 +0000 UTC m=+1429.964489394"
Oct 30 19:28:02.896024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:02.895991    4320 generic.go:356] "Generic (PLEG): container finished" podID="729bea98-1e30-4252-96d6-b484a139050d" containerID="cc5f5d7c8c4fd3e6f2dde1b453c779ba4617e8b7b623afe9dbd8095defa45737" exitCode=0
Oct 30 19:28:02.896413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:02.896054    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb" event={"ID":"729bea98-1e30-4252-96d6-b484a139050d","Type":"ContainerDied","Data":"cc5f5d7c8c4fd3e6f2dde1b453c779ba4617e8b7b623afe9dbd8095defa45737"}
Oct 30 19:28:04.074434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.074409    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:04.110632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.110581    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\") pod \"729bea98-1e30-4252-96d6-b484a139050d\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") "
Oct 30 19:28:04.110789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.110679    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\" (UniqueName: \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\") pod \"729bea98-1e30-4252-96d6-b484a139050d\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") "
Oct 30 19:28:04.110789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.110730    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\") pod \"729bea98-1e30-4252-96d6-b484a139050d\" (UID: \"729bea98-1e30-4252-96d6-b484a139050d\") "
Oct 30 19:28:04.111115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.111085    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca" pod "729bea98-1e30-4252-96d6-b484a139050d" (UID: "729bea98-1e30-4252-96d6-b484a139050d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:04.111297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.111275    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729bea98-1e30-4252-96d6-b484a139050d-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:04.112572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.112541    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca" pod "729bea98-1e30-4252-96d6-b484a139050d" (UID: "729bea98-1e30-4252-96d6-b484a139050d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:04.112885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.112865    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2" pod "729bea98-1e30-4252-96d6-b484a139050d" (UID: "729bea98-1e30-4252-96d6-b484a139050d"). InnerVolumeSpecName "kube-api-access-m6jz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:04.212712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.212628    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729bea98-1e30-4252-96d6-b484a139050d-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:04.212712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.212658    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jz2\" (UniqueName: \"kubernetes.io/projected/729bea98-1e30-4252-96d6-b484a139050d-kube-api-access-m6jz2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:04.910490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.910459    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"
Oct 30 19:28:04.910666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.910485    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb" event={"ID":"729bea98-1e30-4252-96d6-b484a139050d","Type":"ContainerDied","Data":"3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be"}
Oct 30 19:28:04.910666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:04.910544    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3171eb7b51af985cef095a31419993c9f03b978de75c40f81f70b77c50ccf9be"
Oct 30 19:28:12.347771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.347738    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"]
Oct 30 19:28:12.383620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.349501    4320 cpu_manager.go:414] "containerMap: removing container" podUID="729bea98-1e30-4252-96d6-b484a139050d" containerName="revoke-keys"
Oct 30 19:28:12.386681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.386586    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729bea98-1e30-4252-96d6-b484a139050d" containerName="revoke-keys"
Oct 30 19:28:12.432620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.387084    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="729bea98-1e30-4252-96d6-b484a139050d" containerName="revoke-keys"
Oct 30 19:28:12.459393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.459363    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.494888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.494860    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"]
Oct 30 19:28:12.510619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.507340    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:12.510619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.507475    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:28:12.519312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.519272    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.519751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.519699    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.520099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.520077    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5v29\" (UniqueName: \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.603681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.595321    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:28:12.645481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.645442    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.650028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.649999    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.650235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.650216    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5v29\" (UniqueName: \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.669628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.665015    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.713623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.702267    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5v29\" (UniqueName: \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.759620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.759315    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44027af7-plgt4\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:12.931621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:12.923922    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:13.206751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.170404    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.206751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.179374    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:28:13.263156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.262041    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.288910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.288870    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.310664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.305906    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.325309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.323694    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.599969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.598688    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.599969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.599322    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:28:13.694568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.694526    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:13.903793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:13.902106    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:28:14.201680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.196568    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:14.346622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.345094    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:28:14.392837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.392783    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:14.460745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.458071    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:28:14.747108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.502583    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"]
Oct 30 19:28:14.747700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.531229    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:14.747979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.653136    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:28:14.844537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:14.844503    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:28:15.108053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.095675    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:15.108053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.105343    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:15.209840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.201487    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:15.209840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.208224    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:28:15.283990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.275386    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:15.447150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.360986    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-051fbf61-glqg2"]
Oct 30 19:28:15.447150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.431292    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:15.447150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.440907    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4" event={"ID":"f7ed1050-20a5-44fc-ac64-2660961e0ee9","Type":"ContainerStarted","Data":"5fabc8037421269b3524e97ceea95c23388c9d17483b2960da73d2fcd499d6d3"}
Oct 30 19:28:15.457415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.453162    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:28:15.478702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.478678    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:28:15.493050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:15.492978    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"]
Oct 30 19:28:16.097701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:16.097667    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84a6b5f-af96-493d-bf0d-f2789fbf8115" path="/var/lib/kubelet/pods/c84a6b5f-af96-493d-bf0d-f2789fbf8115/volumes"
Oct 30 19:28:16.448471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:16.448437    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4" event={"ID":"f7ed1050-20a5-44fc-ac64-2660961e0ee9","Type":"ContainerStarted","Data":"9f556ee1cdc8d22337b087bf49dd7093cb576fa5e3b639ef875f22a471439e18"}
Oct 30 19:28:16.499806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:16.499758    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4" podStartSLOduration=5.499745542 podStartE2EDuration="5.499745542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:16.485359579 +0000 UTC m=+1444.520872043" watchObservedRunningTime="2026-10-30 19:28:16.499745542 +0000 UTC m=+1444.535258006"
Oct 30 19:28:17.457561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:17.457524    4320 generic.go:356] "Generic (PLEG): container finished" podID="f7ed1050-20a5-44fc-ac64-2660961e0ee9" containerID="9f556ee1cdc8d22337b087bf49dd7093cb576fa5e3b639ef875f22a471439e18" exitCode=0
Oct 30 19:28:17.458068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:17.457639    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4" event={"ID":"f7ed1050-20a5-44fc-ac64-2660961e0ee9","Type":"ContainerDied","Data":"9f556ee1cdc8d22337b087bf49dd7093cb576fa5e3b639ef875f22a471439e18"}
Oct 30 19:28:18.622832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.622806    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:18.717886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.717816    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\") pod \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") "
Oct 30 19:28:18.718012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.717989    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\") pod \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") "
Oct 30 19:28:18.718061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.718048    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\" (UniqueName: \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\") pod \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\" (UID: \"f7ed1050-20a5-44fc-ac64-2660961e0ee9\") "
Oct 30 19:28:18.718332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.718306    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca" pod "f7ed1050-20a5-44fc-ac64-2660961e0ee9" (UID: "f7ed1050-20a5-44fc-ac64-2660961e0ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:18.718528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.718511    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:18.719846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.719815    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca" pod "f7ed1050-20a5-44fc-ac64-2660961e0ee9" (UID: "f7ed1050-20a5-44fc-ac64-2660961e0ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:18.719966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.719947    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29" pod "f7ed1050-20a5-44fc-ac64-2660961e0ee9" (UID: "f7ed1050-20a5-44fc-ac64-2660961e0ee9"). InnerVolumeSpecName "kube-api-access-r5v29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:18.819535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.819508    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5v29\" (UniqueName: \"kubernetes.io/projected/f7ed1050-20a5-44fc-ac64-2660961e0ee9-kube-api-access-r5v29\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:18.819535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:18.819534    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ed1050-20a5-44fc-ac64-2660961e0ee9-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:19.472623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:19.472578    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"
Oct 30 19:28:19.472815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:19.472627    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4" event={"ID":"f7ed1050-20a5-44fc-ac64-2660961e0ee9","Type":"ContainerDied","Data":"5fabc8037421269b3524e97ceea95c23388c9d17483b2960da73d2fcd499d6d3"}
Oct 30 19:28:19.472815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:19.472702    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fabc8037421269b3524e97ceea95c23388c9d17483b2960da73d2fcd499d6d3"
Oct 30 19:28:21.131846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.131813    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 19:28:21.132542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.132523    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f7ed1050-20a5-44fc-ac64-2660961e0ee9" containerName="revoke-keys"
Oct 30 19:28:21.132631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.132545    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ed1050-20a5-44fc-ac64-2660961e0ee9" containerName="revoke-keys"
Oct 30 19:28:21.132710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.132699    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ed1050-20a5-44fc-ac64-2660961e0ee9" containerName="revoke-keys"
Oct 30 19:28:21.136746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.136722    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.158029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.158003    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 19:28:21.164269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.164245    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"distinct-w1-5ac614-kserve-self-signed-certs\""
Oct 30 19:28:21.251677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.251632    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vpd\" (UniqueName: \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.251845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.251828    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.251912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.251857    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.252003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.251987    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.252064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.252047    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.252190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.252167    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.354656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354498    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.354656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354592    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.354920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354672    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.354920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354742    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vpd\" (UniqueName: \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.354920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354913    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.355058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.354939    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.356627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.355855    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.356627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.356101    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.356627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.356327    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.365266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.365240    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.371638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.371174    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.381363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.381334    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vpd\" (UniqueName: \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\") pod \"distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.452522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.452488    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:21.767968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:21.767936    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 19:28:21.768258 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:21.768237    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcac09df1_8a18_487c_a762_7d47949ff198.slice/crio-93aeffd87e676324acdddde5131734626ec91fa4a5927d7b40eae6e1dfe845f5 WatchSource:0}: Error finding container 93aeffd87e676324acdddde5131734626ec91fa4a5927d7b40eae6e1dfe845f5: Status 404 returned error can't find the container with id 93aeffd87e676324acdddde5131734626ec91fa4a5927d7b40eae6e1dfe845f5
Oct 30 19:28:22.507973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:22.507934    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" event={"ID":"cac09df1-8a18-487c-a762-7d47949ff198","Type":"ContainerStarted","Data":"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"}
Oct 30 19:28:22.507973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:22.507976    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" event={"ID":"cac09df1-8a18-487c-a762-7d47949ff198","Type":"ContainerStarted","Data":"93aeffd87e676324acdddde5131734626ec91fa4a5927d7b40eae6e1dfe845f5"}
Oct 30 19:28:22.550017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:22.549972    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podStartSLOduration=1.549959976 podStartE2EDuration="1.549959976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:22.5465376 +0000 UTC m=+1450.582050064" watchObservedRunningTime="2026-10-30 19:28:22.549959976 +0000 UTC m=+1450.585472490"
Oct 30 19:28:23.453331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:23.453300    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:24.029332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:24.029302    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"]
Oct 30 19:28:24.041378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:24.041350    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7d0d52-jzgf6"]
Oct 30 19:28:24.097663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:24.097631    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eab5c6-62c1-44ca-85c5-7a8585a3e52b" path="/var/lib/kubelet/pods/b4eab5c6-62c1-44ca-85c5-7a8585a3e52b/volumes"
Oct 30 19:28:28.645450 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:28:28.645405    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:28:31.254674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.254642    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:28:31.263237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.263212    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.311809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.311775    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:28:31.388438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388400    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388472    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388554    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388640    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388723    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388769    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388825    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388847    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhn2f\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.388872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.388866    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.453910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.453881    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:31.466502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.466481    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:31.490509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490479    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490566    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490624    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490677    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490771    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490833    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490873    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490901    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhn2f\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.490995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.490932    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.491217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.491195    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.491304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.491258    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.491397 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.491374    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.491452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.491394    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.491452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.491405    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.493163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.493140    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.493258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.493163    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.511824 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.511773    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhn2f\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.511967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.511946    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.581525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.581497    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:31.586136 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.586117    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:31.916948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.916921    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:28:31.917975 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:31.917945    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733aef98_88e7_41d7_92b3_ae12b8090371.slice/crio-3e58503cbd48bc9eab1e2cef9cc1bb9f6c4dab2ddf6970cd23c71f21e6a7607a WatchSource:0}: Error finding container 3e58503cbd48bc9eab1e2cef9cc1bb9f6c4dab2ddf6970cd23c71f21e6a7607a: Status 404 returned error can't find the container with id 3e58503cbd48bc9eab1e2cef9cc1bb9f6c4dab2ddf6970cd23c71f21e6a7607a
Oct 30 19:28:31.920629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.920578    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:31.920709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.920663    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:31.920709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:31.920692    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:32.582820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:32.582787    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" event={"ID":"733aef98-88e7-41d7-92b3-ae12b8090371","Type":"ContainerStarted","Data":"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"}
Oct 30 19:28:32.583270 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:32.582920    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" event={"ID":"733aef98-88e7-41d7-92b3-ae12b8090371","Type":"ContainerStarted","Data":"3e58503cbd48bc9eab1e2cef9cc1bb9f6c4dab2ddf6970cd23c71f21e6a7607a"}
Oct 30 19:28:32.632997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:32.632945    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podStartSLOduration=1.6329296279999999 podStartE2EDuration="1.632929628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:32.629270979 +0000 UTC m=+1460.664783434" watchObservedRunningTime="2026-10-30 19:28:32.632929628 +0000 UTC m=+1460.668442092"
Oct 30 19:28:33.593650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:33.589041    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:33.595657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:33.595503    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:33.595657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:33.595563    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:34.588621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:34.585920    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:34.588621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:34.585981    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:35.100460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.100413    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:28:35.593624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.592686    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:35.593624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.592745    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:35.765306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.765271    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"]
Oct 30 19:28:35.785712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.782903    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.799476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.799444    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:35.808857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.807106    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"]
Oct 30 19:28:35.811714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.811674    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:28:35.854187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.854065    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qk7\" (UniqueName: \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.855765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.855731    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.855895 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.855815    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.957335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.957271    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qk7\" (UniqueName: \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.957499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.957374    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.957499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.957431    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.958384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.958294    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:35.960326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:35.960303    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:36.003653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:36.003573    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qk7\" (UniqueName: \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\") pod \"maas-api-revoke-sub-b56bd5d8-xgzf8\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:36.122010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:36.121932    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:36.793633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:36.778814    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"]
Oct 30 19:28:36.838275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:36.807722    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:36.838275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:36.807782    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:36.841138 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:36.841111    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7206a119_8d87_4cd4_b7d0_43f8284f8180.slice/crio-8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7 WatchSource:0}: Error finding container 8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7: Status 404 returned error can't find the container with id 8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7
Oct 30 19:28:37.253620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.252692    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:37.606623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.605434    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:37.606623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.605514    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:37.632379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.632332    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:28:37.740740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.740695    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:37.808680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.805220    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8" event={"ID":"7206a119-8d87-4cd4-b7d0-43f8284f8180","Type":"ContainerStarted","Data":"10c2cb2197204f70d37b51fa6723a4c9f8810f00d811ae5ccd1ee68c902f5a98"}
Oct 30 19:28:37.808680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.805302    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8" event={"ID":"7206a119-8d87-4cd4-b7d0-43f8284f8180","Type":"ContainerStarted","Data":"8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7"}
Oct 30 19:28:37.823718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.823677    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Oct 30 19:28:37.877683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.877620    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8" podStartSLOduration=2.8775801420000002 podStartE2EDuration="2.877580142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:37.848907069 +0000 UTC m=+1465.884419533" watchObservedRunningTime="2026-10-30 19:28:37.877580142 +0000 UTC m=+1465.913092608"
Oct 30 19:28:37.934804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:37.934747    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.105864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.105754    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:28:38.125632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.117740    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.225622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.218025    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.228828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.228791    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.257778 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.257730    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.258643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.258466    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:28:38.318086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.317715    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:28:38.517109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.516686    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:38.548678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.533675    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.610643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.595580    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:38.610643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.595673    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:38.653068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.650240    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:28:38.653068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.650627    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:38.654258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.654230    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:38.654395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.654279    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.762620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.761979    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:28:38.797622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.762900    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:28:38.950571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.949697    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.950571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:38.950094    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:28:39.169620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.144797    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:28:39.169620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.168849    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:28:39.169954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.169925    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Oct 30 19:28:39.180336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.180242    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:28:39.181374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.181350    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:28:39.230614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.229702    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.230614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.230096    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:28:39.262657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.260448    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:28:39.295645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.287485    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.295645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.287940    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:28:39.464684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.370765    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.465273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.465249    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:28:39.490772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.490683    4320 generic.go:356] "Generic (PLEG): container finished" podID="7206a119-8d87-4cd4-b7d0-43f8284f8180" containerID="10c2cb2197204f70d37b51fa6723a4c9f8810f00d811ae5ccd1ee68c902f5a98" exitCode=0
Oct 30 19:28:39.490772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.490760    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8" event={"ID":"7206a119-8d87-4cd4-b7d0-43f8284f8180","Type":"ContainerDied","Data":"10c2cb2197204f70d37b51fa6723a4c9f8810f00d811ae5ccd1ee68c902f5a98"}
Oct 30 19:28:39.523513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.523473    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.524146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.524118    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:28:39.541326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.541286    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:28:39.541887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.541863    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:28:39.633692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.633656    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:39.634010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.633964    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:39.661548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.660665    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.709770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.709735    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:39.710066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.710024    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.760810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.760686    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.777912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.776028    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.867660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.867312    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:39.867660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.867374    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.907788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.907752    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:39.908031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.908000    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:39.950870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:39.950825    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.165626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.160441    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:28:40.247744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.192686    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:40.248059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.248030    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.248237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.246990    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.248885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.192754    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.264017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.263716    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.291714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.291674    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.472672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.465069    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.515775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.515730    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.544791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.544715    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.572129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.568461    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:28:40.602622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.599885    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:40.602622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.599946    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:40.618102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.616717    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.683855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.683813    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.740838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.740727    4320 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:40.741126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.741075    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:40.800831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.800793    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:28:40.920634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.911125    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:28:40.950873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.950448    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:28:40.952419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:40.951769    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:28:41.073046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.072961    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:28:41.075388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.075367    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:28:41.106052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.105863    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:28:41.406563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.406521    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:28:41.408219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.408188    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:41.420445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.420424    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:28:41.561196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.561164    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:28:41.575085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.575048    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:41.635924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.635883    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:41.636088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.635948    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:41.835673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.814755    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:41.987168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:41.987135    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:42.194195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.193913    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\") pod \"7206a119-8d87-4cd4-b7d0-43f8284f8180\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") "
Oct 30 19:28:42.194195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.194099    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\" (UniqueName: \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\") pod \"7206a119-8d87-4cd4-b7d0-43f8284f8180\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") "
Oct 30 19:28:42.194195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.194185    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\") pod \"7206a119-8d87-4cd4-b7d0-43f8284f8180\" (UID: \"7206a119-8d87-4cd4-b7d0-43f8284f8180\") "
Oct 30 19:28:42.196861 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.196832    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca" pod "7206a119-8d87-4cd4-b7d0-43f8284f8180" (UID: "7206a119-8d87-4cd4-b7d0-43f8284f8180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:42.266659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.264767    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7" pod "7206a119-8d87-4cd4-b7d0-43f8284f8180" (UID: "7206a119-8d87-4cd4-b7d0-43f8284f8180"). InnerVolumeSpecName "kube-api-access-b9qk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:42.266659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.264847    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca" pod "7206a119-8d87-4cd4-b7d0-43f8284f8180" (UID: "7206a119-8d87-4cd4-b7d0-43f8284f8180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:42.300621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.298267    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:42.300621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.298302    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qk7\" (UniqueName: \"kubernetes.io/projected/7206a119-8d87-4cd4-b7d0-43f8284f8180-kube-api-access-b9qk7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:42.300621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.298322    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7206a119-8d87-4cd4-b7d0-43f8284f8180-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:42.350061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.350034    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:28:42.350328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.350314    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 19:28:42.400626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.400365    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:28:42.414472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.413402    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:42.429999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.429967    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:28:42.581840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.581760    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:28:42.581840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.581808    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:28:42.658930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.658904    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"
Oct 30 19:28:42.659106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.658947    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8" event={"ID":"7206a119-8d87-4cd4-b7d0-43f8284f8180","Type":"ContainerDied","Data":"8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7"}
Oct 30 19:28:42.659106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:42.658991    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8588beec88f0f43ca799952250ac75e43819af90976b79d616dd10a61ee9b5a7"
Oct 30 19:28:43.586923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:43.586892    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:43.587338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:43.587318    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:43.588038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:43.588021    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:28:46.215704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.215668    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"]
Oct 30 19:28:46.216338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.216320    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7206a119-8d87-4cd4-b7d0-43f8284f8180" containerName="revoke-keys"
Oct 30 19:28:46.216400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.216340    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7206a119-8d87-4cd4-b7d0-43f8284f8180" containerName="revoke-keys"
Oct 30 19:28:46.216487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.216464    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7206a119-8d87-4cd4-b7d0-43f8284f8180" containerName="revoke-keys"
Oct 30 19:28:46.220238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.220220    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.240009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.239990    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"]
Oct 30 19:28:46.247841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.247814    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:46.247933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.247894    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:28:46.258806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.258781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnhb\" (UniqueName: \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.258914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.258854    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.259016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.259000    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.288068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.288044    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:28:46.293640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.293620    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.320536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.318700    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:28:46.360747 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.360720    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srs4v\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.360879 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.360777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.360942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.360875    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.361002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.360982    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361041    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361095    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361181    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361214    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361273    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnhb\" (UniqueName: \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.361355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361301    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.361538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361517    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.361830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.361806    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.362387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.362362    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.363565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.363544    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.381329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.381308    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnhb\" (UniqueName: \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\") pod \"maas-api-revoke-sub-2e147182-lwmb9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.463292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463273    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs4v\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463305    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463555    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463644    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463681    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463738    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463759    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463807    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.463955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.463917    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.464082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.464038    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.464153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.464097    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.464216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.464168    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.464387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.464360    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.464648 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.464586    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.465799 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.465751    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.465894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.465861    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.482958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.482934    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.483057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.482994    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs4v\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.534123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.534098    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:46.612268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.612193    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:46.882828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:46.879531    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"]
Oct 30 19:28:46.893231 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:46.893192    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b238ee1_c6b9_45da_b912_2d803abad2d9.slice/crio-25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94 WatchSource:0}: Error finding container 25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94: Status 404 returned error can't find the container with id 25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94
Oct 30 19:28:47.006793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.006768    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:28:47.006905 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:28:47.006877    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bf9b9f_a56e_4da7_8290_622c96e5216e.slice/crio-f756227d844a17e8537523c8bb11cbbcf7736ab20d1a24b73316f014031f44c7 WatchSource:0}: Error finding container f756227d844a17e8537523c8bb11cbbcf7736ab20d1a24b73316f014031f44c7: Status 404 returned error can't find the container with id f756227d844a17e8537523c8bb11cbbcf7736ab20d1a24b73316f014031f44c7
Oct 30 19:28:47.009534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.009504    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:47.009616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.009565    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:47.009654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.009628    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:47.707591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.707549    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" event={"ID":"a9bf9b9f-a56e-4da7-8290-622c96e5216e","Type":"ContainerStarted","Data":"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"}
Oct 30 19:28:47.708091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.707633    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" event={"ID":"a9bf9b9f-a56e-4da7-8290-622c96e5216e","Type":"ContainerStarted","Data":"f756227d844a17e8537523c8bb11cbbcf7736ab20d1a24b73316f014031f44c7"}
Oct 30 19:28:47.710523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.710491    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9" event={"ID":"2b238ee1-c6b9-45da-b912-2d803abad2d9","Type":"ContainerStarted","Data":"1589d439f021e17fab54af115ab5e6b07079a29df8db7bb657e75998f48af0ba"}
Oct 30 19:28:47.710677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.710549    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9" event={"ID":"2b238ee1-c6b9-45da-b912-2d803abad2d9","Type":"ContainerStarted","Data":"25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94"}
Oct 30 19:28:47.780620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.778532    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podStartSLOduration=1.778504736 podStartE2EDuration="1.778504736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:47.761903318 +0000 UTC m=+1475.797415777" watchObservedRunningTime="2026-10-30 19:28:47.778504736 +0000 UTC m=+1475.814017203"
Oct 30 19:28:47.814377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:47.814316    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9" podStartSLOduration=1.814301859 podStartE2EDuration="1.814301859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:47.807222955 +0000 UTC m=+1475.842735420" watchObservedRunningTime="2026-10-30 19:28:47.814301859 +0000 UTC m=+1475.849814314"
Oct 30 19:28:48.613631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:48.613560    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:48.615373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:48.615345    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:48.615509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:48.615393    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:48.718691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:48.718657    4320 generic.go:356] "Generic (PLEG): container finished" podID="2b238ee1-c6b9-45da-b912-2d803abad2d9" containerID="1589d439f021e17fab54af115ab5e6b07079a29df8db7bb657e75998f48af0ba" exitCode=0
Oct 30 19:28:48.719072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:48.718746    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9" event={"ID":"2b238ee1-c6b9-45da-b912-2d803abad2d9","Type":"ContainerDied","Data":"1589d439f021e17fab54af115ab5e6b07079a29df8db7bb657e75998f48af0ba"}
Oct 30 19:28:49.614056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:49.614020    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:49.614232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:49.614080    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:49.934026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:49.934001    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:50.031381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.031344    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\") pod \"2b238ee1-c6b9-45da-b912-2d803abad2d9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") "
Oct 30 19:28:50.031639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.031608    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\" (UniqueName: \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\") pod \"2b238ee1-c6b9-45da-b912-2d803abad2d9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") "
Oct 30 19:28:50.031737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.031651    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\") pod \"2b238ee1-c6b9-45da-b912-2d803abad2d9\" (UID: \"2b238ee1-c6b9-45da-b912-2d803abad2d9\") "
Oct 30 19:28:50.031808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.031741    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca" pod "2b238ee1-c6b9-45da-b912-2d803abad2d9" (UID: "2b238ee1-c6b9-45da-b912-2d803abad2d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:50.032628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.032536    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:50.034182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.034150    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca" pod "2b238ee1-c6b9-45da-b912-2d803abad2d9" (UID: "2b238ee1-c6b9-45da-b912-2d803abad2d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:50.034476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.034451    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb" pod "2b238ee1-c6b9-45da-b912-2d803abad2d9" (UID: "2b238ee1-c6b9-45da-b912-2d803abad2d9"). InnerVolumeSpecName "kube-api-access-4hnhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:50.133556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.133521    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnhb\" (UniqueName: \"kubernetes.io/projected/2b238ee1-c6b9-45da-b912-2d803abad2d9-kube-api-access-4hnhb\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:50.133556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.133553    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b238ee1-c6b9-45da-b912-2d803abad2d9-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:28:50.613277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.613242    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:50.613428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.613291    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:50.734242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.734217    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"
Oct 30 19:28:50.734378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.734260    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9" event={"ID":"2b238ee1-c6b9-45da-b912-2d803abad2d9","Type":"ContainerDied","Data":"25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94"}
Oct 30 19:28:50.734378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:50.734308    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25137ea54c4aed58a91ba8cb8f4a6995bf611288bce770763c4b251a8c7b4f94"
Oct 30 19:28:51.032506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:51.032469    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"]
Oct 30 19:28:51.046005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:51.045975    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8a71bd-tznff"]
Oct 30 19:28:51.613895 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:51.613854    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:51.614073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:51.613923    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:52.109365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:52.109330    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133d1ec5-66c7-4be4-8981-9bfb401a6d65" path="/var/lib/kubelet/pods/133d1ec5-66c7-4be4-8981-9bfb401a6d65/volumes"
Oct 30 19:28:52.613512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:52.613476    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:52.613804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:52.613537    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:53.346665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.345638    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:28:53.346665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.346547    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2b238ee1-c6b9-45da-b912-2d803abad2d9" containerName="revoke-keys"
Oct 30 19:28:53.346665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.346562    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b238ee1-c6b9-45da-b912-2d803abad2d9" containerName="revoke-keys"
Oct 30 19:28:53.347326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.346738    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b238ee1-c6b9-45da-b912-2d803abad2d9" containerName="revoke-keys"
Oct 30 19:28:53.360668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.352547    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.379503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.377390    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c5e08e43-openshift-default-dockercfg-kdb8z\""
Oct 30 19:28:53.419021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.418991    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:28:53.455875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.455840    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.455963    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456000    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbw2\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456036    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456093    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456170    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456205    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456311    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.456434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.456357    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.557896    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.557952    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558058    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558094    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558212    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558311    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558346    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbw2\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558383    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.558631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558437    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.559233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.558942    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.559233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.559197    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.570628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.560538    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.570628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.561791    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.570628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.562142    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.604518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.595286    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbw2\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.613625 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.610140    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.616953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.616161    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.616953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.616492    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:53.616953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.616541    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:28:53.617301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.617252    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\") pod \"e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") " pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:53.676661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:53.675968    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:54.123513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.121850    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:28:54.125329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.125275    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:54.125443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.125375    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:54.125443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.125418    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:28:54.664881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.664847    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:54.665360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.665336    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:54.666371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.666352    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:28:54.966465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.966200    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerStarted","Data":"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"}
Oct 30 19:28:54.966465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:54.966298    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerStarted","Data":"e155155a9abdadf1202e20bf2477a409e88f65dc91f21882967a1c63e023239a"}
Oct 30 19:28:55.022464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:55.022385    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podStartSLOduration=2.022358923 podStartE2EDuration="2.022358923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:55.010740742 +0000 UTC m=+1483.046253207" watchObservedRunningTime="2026-10-30 19:28:55.022358923 +0000 UTC m=+1483.057871390"
Oct 30 19:28:55.678297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:55.678265    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:55.683448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:55.683418    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:55.974480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:55.974405    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:55.975165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:28:55.975149    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:28:58.645130 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:28:58.645097    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:29:00.545826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.545791    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"]
Oct 30 19:29:00.628179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.628153    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.646501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.646454    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.646728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.646705    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.646931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.646911    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6wz\" (UniqueName: \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.665611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.665570    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"]
Oct 30 19:29:00.706316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.706158    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:29:00.706316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.706239    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:00.755465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.755425    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.755639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.755478    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.755639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.755563    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6wz\" (UniqueName: \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.757179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.757151    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.858959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.835727    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:00.920624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:00.894410    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6wz\" (UniqueName: \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\") pod \"maas-api-revoke-sub-63bf930a-qrsp6\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:01.026186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.025929    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:01.586622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.586317    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:29:01.608907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.608876    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.644622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.641156    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.768835    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.768953    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.768990    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769049    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769080    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg48\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769191    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769292    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769328    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.782974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.769362    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.976037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.976001    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.990001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.989975    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.990563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.990543    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.990862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.990840    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.991683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.991662    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.991860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.991841    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.992018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.992002    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.992146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.992129    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg48\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:01.992368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.992350    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.992742    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.990478    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.992870    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.981286    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:01.991508    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.030054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.027339    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg48\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.104077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.095065    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.121637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.112412    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:29:02.121637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.114542    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:29:02.159462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.158451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.281502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.281427    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.300447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.300403    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:02.338026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.337195    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:29:02.344258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.344229    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:02.442890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.442842    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:02.558688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.545760    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded"
Oct 30 19:29:02.583311 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.583267    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:02.621532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.621499    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"]
Oct 30 19:29:02.622132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.622101    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6" event={"ID":"13b2a10f-9078-4cb1-8de3-987d44848151","Type":"ContainerStarted","Data":"b4bf952cf4faafb074e550e9c655b9f779b48d12efd38c9d2e04db9f727e92ab"}
Oct 30 19:29:02.759635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.739762    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:02.971694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.970080    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:02.977698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:02.977660    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:03.088740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.088691    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:03.114723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.114677    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:29:03.119651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.119578    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:29:03.165140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.144950    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:29:03.165140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.145275    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:29:03.165140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.145311    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"]
Oct 30 19:29:03.165140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.152024    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9bc1a7d-dbpbj"]
Oct 30 19:29:03.425519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.425419    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:29:03.454796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.440397    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:29:03.505622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.492649    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:29:03.505622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.492702    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:03.575732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.575179    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:29:03.610669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.580498    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:29:03.682688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.678707    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:03.682688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.679432    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:29:03.759634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.753647    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:29:03.759634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:03.754106    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:29:04.070052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.069993    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:29:04.096842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.096810    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:29:04.141108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.141057    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.142703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.142669    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.162061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.151782    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:29:04.226621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.223454    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.236619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.234527    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.454750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.454710    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:29:04.455377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.429780    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:29:04.577627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.577036    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.577627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.577381    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:29:04.713703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.712292    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:04.769494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:04.769453    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.074817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.074696    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.165583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.165507    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2732b174-6f99-4ba9-aec3-e0fdc4c1936d" path="/var/lib/kubelet/pods/2732b174-6f99-4ba9-aec3-e0fdc4c1936d/volumes"
Oct 30 19:29:05.173650 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:29:05.173617    4320 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.076s"
Oct 30 19:29:05.173864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.173850    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:29:05.174830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.174814    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:29:05.174984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.174973    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 19:29:05.175101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.175091    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:29:05.177355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.169836    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.183757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.183731    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.218204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.210239    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:05.218204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.210328    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:05.218204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.210385    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:05.218204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.213705    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:29:05.218204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.214981    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6" event={"ID":"13b2a10f-9078-4cb1-8de3-987d44848151","Type":"ContainerStarted","Data":"091d36fb30e27cf25436867da8fcce3eaf6d44814af80c27f81826d772bd68a6"}
Oct 30 19:29:05.244701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.244668    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:29:05.296641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.293047    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 19:29:05.300581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.300424    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 19:29:05.427340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.427285    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.447830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.435444    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 19:29:05.462148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.462112    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.463337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.463311    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 19:29:05.475742 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.475700    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:05.476073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.476054    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:29:05.500459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.484947    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 19:29:05.525798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.525774    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:29:05.561226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:05.561204    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 19:29:06.227983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.227942    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" event={"ID":"3d29bea6-5523-4842-ab2d-6d719147ccf4","Type":"ContainerStarted","Data":"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"}
Oct 30 19:29:06.228480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.228013    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" event={"ID":"3d29bea6-5523-4842-ab2d-6d719147ccf4","Type":"ContainerStarted","Data":"cf5d70b2bf443ec5ad37f9206cd44768de2484458f39aac1269260653531d466"}
Oct 30 19:29:06.230204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.230171    4320 generic.go:356] "Generic (PLEG): container finished" podID="13b2a10f-9078-4cb1-8de3-987d44848151" containerID="091d36fb30e27cf25436867da8fcce3eaf6d44814af80c27f81826d772bd68a6" exitCode=0
Oct 30 19:29:06.230351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.230271    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6" event={"ID":"13b2a10f-9078-4cb1-8de3-987d44848151","Type":"ContainerDied","Data":"091d36fb30e27cf25436867da8fcce3eaf6d44814af80c27f81826d772bd68a6"}
Oct 30 19:29:06.306117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.306069    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podStartSLOduration=5.30605379 podStartE2EDuration="5.30605379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:06.299127739 +0000 UTC m=+1494.334640204" watchObservedRunningTime="2026-10-30 19:29:06.30605379 +0000 UTC m=+1494.341566257"
Oct 30 19:29:06.355096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.355062    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:06.356199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.356175    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:06.356331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:06.356219    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:07.355193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.355163    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:07.355651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.355304    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:07.503559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.503495    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:07.565705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.565676    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\") pod \"13b2a10f-9078-4cb1-8de3-987d44848151\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") "
Oct 30 19:29:07.565890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.565790    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\") pod \"13b2a10f-9078-4cb1-8de3-987d44848151\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") "
Oct 30 19:29:07.565890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.565864    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\" (UniqueName: \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\") pod \"13b2a10f-9078-4cb1-8de3-987d44848151\" (UID: \"13b2a10f-9078-4cb1-8de3-987d44848151\") "
Oct 30 19:29:07.566099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.566040    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca" pod "13b2a10f-9078-4cb1-8de3-987d44848151" (UID: "13b2a10f-9078-4cb1-8de3-987d44848151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:07.566996 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.566968    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:07.568201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.568178    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz" pod "13b2a10f-9078-4cb1-8de3-987d44848151" (UID: "13b2a10f-9078-4cb1-8de3-987d44848151"). InnerVolumeSpecName "kube-api-access-9x6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:07.578658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.568639    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca" pod "13b2a10f-9078-4cb1-8de3-987d44848151" (UID: "13b2a10f-9078-4cb1-8de3-987d44848151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:07.668092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.668064    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2a10f-9078-4cb1-8de3-987d44848151-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:07.668092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:07.668091    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6wz\" (UniqueName: \"kubernetes.io/projected/13b2a10f-9078-4cb1-8de3-987d44848151-kube-api-access-9x6wz\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:08.248260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:08.248234    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"
Oct 30 19:29:08.248439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:08.248257    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6" event={"ID":"13b2a10f-9078-4cb1-8de3-987d44848151","Type":"ContainerDied","Data":"b4bf952cf4faafb074e550e9c655b9f779b48d12efd38c9d2e04db9f727e92ab"}
Oct 30 19:29:08.248439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:08.248301    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4bf952cf4faafb074e550e9c655b9f779b48d12efd38c9d2e04db9f727e92ab"
Oct 30 19:29:08.355199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:08.355162    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:08.355666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:08.355220    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:09.251974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.251943    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:29:09.252851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.252830    4320 cpu_manager.go:414] "containerMap: removing container" podUID="13b2a10f-9078-4cb1-8de3-987d44848151" containerName="revoke-keys"
Oct 30 19:29:09.252973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.252963    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b2a10f-9078-4cb1-8de3-987d44848151" containerName="revoke-keys"
Oct 30 19:29:09.253214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.253203    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b2a10f-9078-4cb1-8de3-987d44848151" containerName="revoke-keys"
Oct 30 19:29:09.279624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.257154    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.319664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.317829    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:29:09.331620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.331559    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c5e08e43-dockercfg-sr8lg\""
Oct 30 19:29:09.331907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.331887    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c5e08e43\""
Oct 30 19:29:09.340627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.336586    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.341015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.340991    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzqw\" (UniqueName: \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.365673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.362298    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:09.365673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.362357    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:09.452262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.445635    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.452833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.452800    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzqw\" (UniqueName: \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.453556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.447982    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.521235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.518194    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzqw\" (UniqueName: \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\") pod \"payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") " pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:09.640147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:09.640112    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:10.008933 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:29:10.008879    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cd6042_1a5a_4c21_adff_279ee94e76d3.slice/crio-50ceab5ba7c9e1056d8e8001edd0703a60ea9eed3714a5419ddc30f1e504a095 WatchSource:0}: Error finding container 50ceab5ba7c9e1056d8e8001edd0703a60ea9eed3714a5419ddc30f1e504a095: Status 404 returned error can't find the container with id 50ceab5ba7c9e1056d8e8001edd0703a60ea9eed3714a5419ddc30f1e504a095
Oct 30 19:29:10.016617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.013778    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:29:10.267253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.267158    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" event={"ID":"36cd6042-1a5a-4c21-adff-279ee94e76d3","Type":"ContainerStarted","Data":"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"}
Oct 30 19:29:10.267253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.267224    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" event={"ID":"36cd6042-1a5a-4c21-adff-279ee94e76d3","Type":"ContainerStarted","Data":"50ceab5ba7c9e1056d8e8001edd0703a60ea9eed3714a5419ddc30f1e504a095"}
Oct 30 19:29:10.267253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.267251    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:10.312195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.312136    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" podStartSLOduration=1.312118585 podStartE2EDuration="1.312118585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:10.30688528 +0000 UTC m=+1498.342397745" watchObservedRunningTime="2026-10-30 19:29:10.312118585 +0000 UTC m=+1498.347631051"
Oct 30 19:29:10.355033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.354999    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:10.355220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:10.355066    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:11.107422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.107149    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 19:29:11.115326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.113582    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 19:29:11.115326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.113711    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.145891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.145861    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"distinct-2-w1-5ac614-kserve-self-signed-certs\""
Oct 30 19:29:11.206646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.206610    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.207841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.207816    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.208190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.208169    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.208938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.208920    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.209725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.209328    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp69h\" (UniqueName: \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.209964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.209948    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.320560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.320519    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.321493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.321467    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.347293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.347215    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.350901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.343814    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.355357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.355316    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.355537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.355419    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.355537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.355522    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.355757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.355610    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.355757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.355668    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp69h\" (UniqueName: \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.356591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.356564    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.356986 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.356965    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.358134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.357696    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:11.358134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.357757    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:11.428792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.428313    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp69h\" (UniqueName: \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\") pod \"distinct-2-w1-5ac614-kserve-666c989d65-mdn4m\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") " pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.433644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.433153    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:11.834031 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:29:11.833990    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30587b0d_6cad_4690_8daa_03b6f2ccbfc5.slice/crio-3b020ced5feb04ca5604e7cd98892a920ffd837a3e6de3539f0e5f760b3de334 WatchSource:0}: Error finding container 3b020ced5feb04ca5604e7cd98892a920ffd837a3e6de3539f0e5f760b3de334: Status 404 returned error can't find the container with id 3b020ced5feb04ca5604e7cd98892a920ffd837a3e6de3539f0e5f760b3de334
Oct 30 19:29:11.834549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:11.834526    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 19:29:12.339214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.339179    4320 scope.go:122] "RemoveContainer" containerID="c4b6c5d178465d8ddf6c55aa139f48afa1c67c63f93da7b49c77382a2ab6398e"
Oct 30 19:29:12.355394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.355356    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:29:12.355635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.355431    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:29:12.413203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.413165    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" event={"ID":"30587b0d-6cad-4690-8daa-03b6f2ccbfc5","Type":"ContainerStarted","Data":"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"}
Oct 30 19:29:12.413428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.413217    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" event={"ID":"30587b0d-6cad-4690-8daa-03b6f2ccbfc5","Type":"ContainerStarted","Data":"3b020ced5feb04ca5604e7cd98892a920ffd837a3e6de3539f0e5f760b3de334"}
Oct 30 19:29:12.433983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.433939    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:12.456517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:12.456447    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" podStartSLOduration=1.456428237 podStartE2EDuration="1.456428237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:12.453042569 +0000 UTC m=+1500.488555034" watchObservedRunningTime="2026-10-30 19:29:12.456428237 +0000 UTC m=+1500.491940702"
Oct 30 19:29:13.359122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:13.359094    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:13.359549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:13.359531    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:13.360234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:13.360216    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:29:16.030312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:16.030284    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"]
Oct 30 19:29:16.042019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:16.041991    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2603c1eb-wzbm6"]
Oct 30 19:29:16.098913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:16.098887    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383f2689-f556-4951-b3ec-5394b5273267" path="/var/lib/kubelet/pods/383f2689-f556-4951-b3ec-5394b5273267/volumes"
Oct 30 19:29:16.181959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:16.181936    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:29:19.610622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:19.607737    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:29:19.610622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:19.607809    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:20.386505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:20.374679    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:20.803677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:20.802709    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-cnvpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:29:20.803677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:20.802776    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:21.403690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:21.403663    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:29:21.434310 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:21.434274    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:21.452199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:21.452176    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:21.568516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:21.568480    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 19:29:22.051920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.051894    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"]
Oct 30 19:29:22.057913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.057896    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.073999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.073972    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"]
Oct 30 19:29:22.159267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.159242    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:22.159411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.159393    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:29:22.193225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.193196    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69z5f\" (UniqueName: \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.193395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.193345    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.193507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.193491    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.294562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.294525    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.294868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.294852    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69z5f\" (UniqueName: \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.295353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.295325    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.295500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.295344    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.312676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.312611    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.317214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.317190    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69z5f\" (UniqueName: \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\") pod \"maas-api-revoke-sub-9b9fe01a-rpzcb\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.374458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.374429    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:22.710250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:22.710219    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"]
Oct 30 19:29:22.711018 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:29:22.710992    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b0fb16_8399_43a9_8876_4f50417a6c10.slice/crio-0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8 WatchSource:0}: Error finding container 0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8: Status 404 returned error can't find the container with id 0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8
Oct 30 19:29:23.585606 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:23.585565    4320 generic.go:356] "Generic (PLEG): container finished" podID="26b0fb16-8399-43a9-8876-4f50417a6c10" containerID="68cd6dbda11dbe6ff9d2da5b884fe65842ba30dd351a014f4df93557f64f3120" exitCode=0
Oct 30 19:29:23.586168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:23.585650    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb" event={"ID":"26b0fb16-8399-43a9-8876-4f50417a6c10","Type":"ContainerDied","Data":"68cd6dbda11dbe6ff9d2da5b884fe65842ba30dd351a014f4df93557f64f3120"}
Oct 30 19:29:23.586168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:23.585698    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb" event={"ID":"26b0fb16-8399-43a9-8876-4f50417a6c10","Type":"ContainerStarted","Data":"0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8"}
Oct 30 19:29:24.742644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.742619    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:24.833017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.832932    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\") pod \"26b0fb16-8399-43a9-8876-4f50417a6c10\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") "
Oct 30 19:29:24.833188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.833167    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\") pod \"26b0fb16-8399-43a9-8876-4f50417a6c10\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") "
Oct 30 19:29:24.833269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.833229    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\" (UniqueName: \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\") pod \"26b0fb16-8399-43a9-8876-4f50417a6c10\" (UID: \"26b0fb16-8399-43a9-8876-4f50417a6c10\") "
Oct 30 19:29:24.833624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.833575    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca" pod "26b0fb16-8399-43a9-8876-4f50417a6c10" (UID: "26b0fb16-8399-43a9-8876-4f50417a6c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:24.833913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.833894    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:24.835686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.835660    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca" pod "26b0fb16-8399-43a9-8876-4f50417a6c10" (UID: "26b0fb16-8399-43a9-8876-4f50417a6c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:24.839746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.839713    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f" pod "26b0fb16-8399-43a9-8876-4f50417a6c10" (UID: "26b0fb16-8399-43a9-8876-4f50417a6c10"). InnerVolumeSpecName "kube-api-access-69z5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:24.935646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.935623    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b0fb16-8399-43a9-8876-4f50417a6c10-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:24.935646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:24.935646    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69z5f\" (UniqueName: \"kubernetes.io/projected/26b0fb16-8399-43a9-8876-4f50417a6c10-kube-api-access-69z5f\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:25.602885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:25.602854    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"
Oct 30 19:29:25.603083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:25.602899    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb" event={"ID":"26b0fb16-8399-43a9-8876-4f50417a6c10","Type":"ContainerDied","Data":"0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8"}
Oct 30 19:29:25.603083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:25.602955    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0981d410ff0ce643f091e736754f14d25fff3eea4686eb024704094b0601c7e8"
Oct 30 19:29:28.650234 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:29:28.650187    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:29:29.030983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:29.030909    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"]
Oct 30 19:29:29.043415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:29.043387    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe10893-m556k"]
Oct 30 19:29:30.108673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:30.108634    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e9d119-6d77-4b77-82fb-52176623f38e" path="/var/lib/kubelet/pods/f2e9d119-6d77-4b77-82fb-52176623f38e/volumes"
Oct 30 19:29:34.149622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.148968    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:29:34.201621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.199791    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 19:29:34.297702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.297658    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:34.320619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.309662    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:34.320619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.313658    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:34.463784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.463689    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:34.577127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:34.577086    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:36.358782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.358744    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"]
Oct 30 19:29:36.359379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.359365    4320 cpu_manager.go:414] "containerMap: removing container" podUID="26b0fb16-8399-43a9-8876-4f50417a6c10" containerName="revoke-keys"
Oct 30 19:29:36.359432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.359380    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b0fb16-8399-43a9-8876-4f50417a6c10" containerName="revoke-keys"
Oct 30 19:29:36.359513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.359504    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b0fb16-8399-43a9-8876-4f50417a6c10" containerName="revoke-keys"
Oct 30 19:29:36.363124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.363107    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.381738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.381716    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"]
Oct 30 19:29:36.388253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.388233    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:36.388365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.388345    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:29:36.459518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.459482    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.460094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.460067    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpv8\" (UniqueName: \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.460250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.460107    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.561690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.561638    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpv8\" (UniqueName: \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.561902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.561704    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.561902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.561874    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.562468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.562446    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.564482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.564451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.582557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.582532    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpv8\" (UniqueName: \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\") pod \"maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.679907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.677977    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:36.985839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:36.985811    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"]
Oct 30 19:29:36.987870 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:29:36.987828    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74990a4b_704d_4169_95f1_5396c2c3bb34.slice/crio-c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313 WatchSource:0}: Error finding container c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313: Status 404 returned error can't find the container with id c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313
Oct 30 19:29:37.843000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:37.842972    4320 generic.go:356] "Generic (PLEG): container finished" podID="74990a4b-704d-4169-95f1-5396c2c3bb34" containerID="7736308a74a54b876e95aab12a7279e1918b3fb086c560ee6b114c1a0e170bea" exitCode=0
Oct 30 19:29:37.843330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:37.843025    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l" event={"ID":"74990a4b-704d-4169-95f1-5396c2c3bb34","Type":"ContainerDied","Data":"7736308a74a54b876e95aab12a7279e1918b3fb086c560ee6b114c1a0e170bea"}
Oct 30 19:29:37.843330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:37.843060    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l" event={"ID":"74990a4b-704d-4169-95f1-5396c2c3bb34","Type":"ContainerStarted","Data":"c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313"}
Oct 30 19:29:39.023926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.023897    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:39.107873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.107781    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\") pod \"74990a4b-704d-4169-95f1-5396c2c3bb34\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") "
Oct 30 19:29:39.107873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.107870    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\" (UniqueName: \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\") pod \"74990a4b-704d-4169-95f1-5396c2c3bb34\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") "
Oct 30 19:29:39.108105 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.107908    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\") pod \"74990a4b-704d-4169-95f1-5396c2c3bb34\" (UID: \"74990a4b-704d-4169-95f1-5396c2c3bb34\") "
Oct 30 19:29:39.108333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.108305    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca" pod "74990a4b-704d-4169-95f1-5396c2c3bb34" (UID: "74990a4b-704d-4169-95f1-5396c2c3bb34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:39.108556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.108536    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:39.110011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.109983    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca" pod "74990a4b-704d-4169-95f1-5396c2c3bb34" (UID: "74990a4b-704d-4169-95f1-5396c2c3bb34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:39.110482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.110464    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8" pod "74990a4b-704d-4169-95f1-5396c2c3bb34" (UID: "74990a4b-704d-4169-95f1-5396c2c3bb34"). InnerVolumeSpecName "kube-api-access-flpv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:39.210469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.210439    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74990a4b-704d-4169-95f1-5396c2c3bb34-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:39.210469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.210467    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flpv8\" (UniqueName: \"kubernetes.io/projected/74990a4b-704d-4169-95f1-5396c2c3bb34-kube-api-access-flpv8\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:39.860650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.860625    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"
Oct 30 19:29:39.860650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.860634    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l" event={"ID":"74990a4b-704d-4169-95f1-5396c2c3bb34","Type":"ContainerDied","Data":"c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313"}
Oct 30 19:29:39.860852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:39.860687    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a63bc7a71aa8f96a933748289908627a4a9dda9fe5516a78d5c741564f2313"
Oct 30 19:29:48.903101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:48.902178    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:29:50.750335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:50.750298    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"]
Oct 30 19:29:50.760163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:50.760051    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4245aa3-wv82l"]
Oct 30 19:29:52.141181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:52.141142    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74990a4b-704d-4169-95f1-5396c2c3bb34" path="/var/lib/kubelet/pods/74990a4b-704d-4169-95f1-5396c2c3bb34/volumes"
Oct 30 19:29:53.903557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.903525    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"]
Oct 30 19:29:53.904856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.904833    4320 cpu_manager.go:414] "containerMap: removing container" podUID="74990a4b-704d-4169-95f1-5396c2c3bb34" containerName="revoke-keys"
Oct 30 19:29:53.904987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.904977    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74990a4b-704d-4169-95f1-5396c2c3bb34" containerName="revoke-keys"
Oct 30 19:29:53.905190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.905181    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="74990a4b-704d-4169-95f1-5396c2c3bb34" containerName="revoke-keys"
Oct 30 19:29:53.912849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.912825    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:53.936062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.936030    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"]
Oct 30 19:29:53.940186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.940158    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:53.940324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:53.940307    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:29:54.057629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.057586    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnzf\" (UniqueName: \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.057837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.057813    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.058010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.057991    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.159426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.159342    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnzf\" (UniqueName: \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.160162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.160134    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.161231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.161210    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.161839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.161033    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.164445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.164393    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.189943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.187541    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnzf\" (UniqueName: \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\") pod \"maas-api-revoke-keys-e2e-ait-c5e08e43-zv548\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.240537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.240509    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:54.622422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:54.622309    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"]
Oct 30 19:29:54.622689 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:29:54.622658    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6491d49_0e1a_4720_8f0a_a852ca811834.slice/crio-1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1 WatchSource:0}: Error finding container 1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1: Status 404 returned error can't find the container with id 1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1
Oct 30 19:29:55.042516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:55.042480    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548" event={"ID":"c6491d49-0e1a-4720-8f0a-a852ca811834","Type":"ContainerStarted","Data":"1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1"}
Oct 30 19:29:56.060078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:56.058492    4320 generic.go:356] "Generic (PLEG): container finished" podID="c6491d49-0e1a-4720-8f0a-a852ca811834" containerID="1805183adb55985fceb5909133eea35509e92f80fb6a8d9dd8519da5b5c6fd45" exitCode=0
Oct 30 19:29:56.060078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:56.058717    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548" event={"ID":"c6491d49-0e1a-4720-8f0a-a852ca811834","Type":"ContainerDied","Data":"1805183adb55985fceb5909133eea35509e92f80fb6a8d9dd8519da5b5c6fd45"}
Oct 30 19:29:57.298635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.298322    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:57.383871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.381138    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\") pod \"c6491d49-0e1a-4720-8f0a-a852ca811834\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") "
Oct 30 19:29:57.383871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.381269    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\" (UniqueName: \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\") pod \"c6491d49-0e1a-4720-8f0a-a852ca811834\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") "
Oct 30 19:29:57.383871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.381444    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\") pod \"c6491d49-0e1a-4720-8f0a-a852ca811834\" (UID: \"c6491d49-0e1a-4720-8f0a-a852ca811834\") "
Oct 30 19:29:57.393920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.393882    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca" pod "c6491d49-0e1a-4720-8f0a-a852ca811834" (UID: "c6491d49-0e1a-4720-8f0a-a852ca811834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:57.396127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.396087    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf" pod "c6491d49-0e1a-4720-8f0a-a852ca811834" (UID: "c6491d49-0e1a-4720-8f0a-a852ca811834"). InnerVolumeSpecName "kube-api-access-7qnzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:57.396309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.396288    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca" pod "c6491d49-0e1a-4720-8f0a-a852ca811834" (UID: "c6491d49-0e1a-4720-8f0a-a852ca811834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:57.493659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.492344    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:57.493844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.493678    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6491d49-0e1a-4720-8f0a-a852ca811834-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:57.493844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:57.493706    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qnzf\" (UniqueName: \"kubernetes.io/projected/c6491d49-0e1a-4720-8f0a-a852ca811834-kube-api-access-7qnzf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:29:58.099558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:58.099527    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"
Oct 30 19:29:58.109872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:58.108585    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548" event={"ID":"c6491d49-0e1a-4720-8f0a-a852ca811834","Type":"ContainerDied","Data":"1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1"}
Oct 30 19:29:58.109872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:29:58.108659    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e38077ec285c4c6f5f4d0017c9ac9288391184bb5c94d1abc67524772f6f9a1"
Oct 30 19:29:58.645216 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:29:58.645175    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:30:00.159467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.159430    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:00.160388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.160358    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c6491d49-0e1a-4720-8f0a-a852ca811834" containerName="revoke-keys"
Oct 30 19:30:00.160388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.160388    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6491d49-0e1a-4720-8f0a-a852ca811834" containerName="revoke-keys"
Oct 30 19:30:00.160684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.160667    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6491d49-0e1a-4720-8f0a-a852ca811834" containerName="revoke-keys"
Oct 30 19:30:00.164419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.164400    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.166004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.165984    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:00.171651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.171630    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.202412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.199507    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:30:00.202412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.199577    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:00.240338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.240303    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:00.244843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.244802    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.245086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.245061    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7pw\" (UniqueName: \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.248831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.248809    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:00.248979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.248919    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.253702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.253660    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:00.260978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.260955    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.303064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.302989    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:00.311199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.311165    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:00.346479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346446    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7pw\" (UniqueName: \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.346652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346530    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvwp\" (UniqueName: \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.346652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346580    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbm6\" (UniqueName: \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.346800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346715    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.346800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346754    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.346897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346864    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.346953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.346904    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vjv\" (UniqueName: \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.347079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.347061    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.347693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.347675    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.370545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.370522    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7pw\" (UniqueName: \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.448582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.448495    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.448582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.448541    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.448841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.448609    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.448841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.448649    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vjv\" (UniqueName: \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.448990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.448962    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvwp\" (UniqueName: \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.449053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.449019    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbm6\" (UniqueName: \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.449460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.449423    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.449622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.449477    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.449622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.449538    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.471853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.471823    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vjv\" (UniqueName: \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\") pod \"maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.471989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.471828    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbm6\" (UniqueName: \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.471989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.471964    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvwp\" (UniqueName: \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.480287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.480264    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:00.493063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.493035    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:00.620570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.616982    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:00.620570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.619568    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:00.829902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.829853    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:00.856114 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:00.855247    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91840ea4_7c00_400d_b306_93c009cd3634.slice/crio-037fde4efdc04207d0fafeaf0772b132d6d567d8836227d9e3507ff7bb46c930 WatchSource:0}: Error finding container 037fde4efdc04207d0fafeaf0772b132d6d567d8836227d9e3507ff7bb46c930: Status 404 returned error can't find the container with id 037fde4efdc04207d0fafeaf0772b132d6d567d8836227d9e3507ff7bb46c930
Oct 30 19:30:00.863942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.863908    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:00.970247 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:00.967998    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182c397d_fe6c_42d8_8d81_21d35144bdba.slice/crio-d7c4708204290592ffaaefab8e95353468a8b52971b8021c15c59177c8a4a14a WatchSource:0}: Error finding container d7c4708204290592ffaaefab8e95353468a8b52971b8021c15c59177c8a4a14a: Status 404 returned error can't find the container with id d7c4708204290592ffaaefab8e95353468a8b52971b8021c15c59177c8a4a14a
Oct 30 19:30:00.970247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.968115    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:00.973758 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:00.972637    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82615a5a_a4f4_4e2f_9167_34e2ef758664.slice/crio-e8ac1dca7d856b2bfddd4c779a515bd18ef5ef0238daacff4e8bf94b8a750114 WatchSource:0}: Error finding container e8ac1dca7d856b2bfddd4c779a515bd18ef5ef0238daacff4e8bf94b8a750114: Status 404 returned error can't find the container with id e8ac1dca7d856b2bfddd4c779a515bd18ef5ef0238daacff4e8bf94b8a750114
Oct 30 19:30:00.973758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:00.972896    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:01.126016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:01.125990    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerStarted","Data":"d80d90c65f4e5188ddb30bb1f3d7e827f857e0be4a2dd2579e28f4ec4f6d6893"}
Oct 30 19:30:01.127764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:01.127742    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerStarted","Data":"037fde4efdc04207d0fafeaf0772b132d6d567d8836227d9e3507ff7bb46c930"}
Oct 30 19:30:01.129079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:01.129057    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerStarted","Data":"d7c4708204290592ffaaefab8e95353468a8b52971b8021c15c59177c8a4a14a"}
Oct 30 19:30:01.130400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:01.130375    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" event={"ID":"82615a5a-a4f4-4e2f-9167-34e2ef758664","Type":"ContainerStarted","Data":"e8ac1dca7d856b2bfddd4c779a515bd18ef5ef0238daacff4e8bf94b8a750114"}
Oct 30 19:30:02.138989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.138959    4320 generic.go:356] "Generic (PLEG): container finished" podID="91840ea4-7c00-400d-b306-93c009cd3634" containerID="d0d4b715d581e4bb7fac1f4b73e317400c4abcbc43f1d60de3b64a5958ee6126" exitCode=60
Oct 30 19:30:02.139380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.139047    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerDied","Data":"d0d4b715d581e4bb7fac1f4b73e317400c4abcbc43f1d60de3b64a5958ee6126"}
Oct 30 19:30:02.139380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.139361    4320 scope.go:122] "RemoveContainer" containerID="d0d4b715d581e4bb7fac1f4b73e317400c4abcbc43f1d60de3b64a5958ee6126"
Oct 30 19:30:02.140999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.140977    4320 generic.go:356] "Generic (PLEG): container finished" podID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerID="2fe2ac921c13a99e8544a8c32f0d73b3c662bbe6c9c2e54a7817889e2c18a96e" exitCode=60
Oct 30 19:30:02.141129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.141048    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerDied","Data":"2fe2ac921c13a99e8544a8c32f0d73b3c662bbe6c9c2e54a7817889e2c18a96e"}
Oct 30 19:30:02.141339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.141323    4320 scope.go:122] "RemoveContainer" containerID="2fe2ac921c13a99e8544a8c32f0d73b3c662bbe6c9c2e54a7817889e2c18a96e"
Oct 30 19:30:02.143279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.143141    4320 generic.go:356] "Generic (PLEG): container finished" podID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerID="fd136b2dc906ad26f4ccd277ff1f543ef7c29db0f41c15f3579e2aa4627a9713" exitCode=60
Oct 30 19:30:02.143345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.143324    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" event={"ID":"82615a5a-a4f4-4e2f-9167-34e2ef758664","Type":"ContainerDied","Data":"fd136b2dc906ad26f4ccd277ff1f543ef7c29db0f41c15f3579e2aa4627a9713"}
Oct 30 19:30:02.143444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.143421    4320 scope.go:122] "RemoveContainer" containerID="fd136b2dc906ad26f4ccd277ff1f543ef7c29db0f41c15f3579e2aa4627a9713"
Oct 30 19:30:02.145700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.145677    4320 generic.go:356] "Generic (PLEG): container finished" podID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerID="df15b53218ab6f538baddd07b29976c07972eb5e402756f03d27bfc0052446bb" exitCode=60
Oct 30 19:30:02.145885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.145726    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerDied","Data":"df15b53218ab6f538baddd07b29976c07972eb5e402756f03d27bfc0052446bb"}
Oct 30 19:30:02.146011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:02.145929    4320 scope.go:122] "RemoveContainer" containerID="df15b53218ab6f538baddd07b29976c07972eb5e402756f03d27bfc0052446bb"
Oct 30 19:30:03.154991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.154958    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerStarted","Data":"163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"}
Oct 30 19:30:03.157063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.157036    4320 generic.go:356] "Generic (PLEG): container finished" podID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerID="4a8f4632bdfcd6ad2d90a224f45679da17491d1e66f748a5828a625c74ad1e74" exitCode=60
Oct 30 19:30:03.157176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.157102    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" event={"ID":"82615a5a-a4f4-4e2f-9167-34e2ef758664","Type":"ContainerDied","Data":"4a8f4632bdfcd6ad2d90a224f45679da17491d1e66f748a5828a625c74ad1e74"}
Oct 30 19:30:03.157176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.157157    4320 scope.go:122] "RemoveContainer" containerID="fd136b2dc906ad26f4ccd277ff1f543ef7c29db0f41c15f3579e2aa4627a9713"
Oct 30 19:30:03.157326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.157307    4320 scope.go:122] "RemoveContainer" containerID="4a8f4632bdfcd6ad2d90a224f45679da17491d1e66f748a5828a625c74ad1e74"
Oct 30 19:30:03.157582 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:03.157560    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8_odh-ai-gateway-infra(82615a5a-a4f4-4e2f-9167-34e2ef758664)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664"
Oct 30 19:30:03.159079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.159061    4320 generic.go:356] "Generic (PLEG): container finished" podID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerID="7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de" exitCode=0
Oct 30 19:30:03.159188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.159120    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerDied","Data":"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"}
Oct 30 19:30:03.159439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.159422    4320 scope.go:122] "RemoveContainer" containerID="7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"
Oct 30 19:30:03.160812 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.160780    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:03.160919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.160867    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:03.160984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.160918    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:03.161414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.161304    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerStarted","Data":"4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"}
Oct 30 19:30:03.163314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.163291    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerStarted","Data":"e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"}
Oct 30 19:30:03.197416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.197374    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" podStartSLOduration=3.197362376 podStartE2EDuration="3.197362376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.195553758 +0000 UTC m=+1551.231066235" watchObservedRunningTime="2026-10-30 19:30:03.197362376 +0000 UTC m=+1551.232874862"
Oct 30 19:30:03.236825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.236779    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" podStartSLOduration=3.236766719 podStartE2EDuration="3.236766719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.233998175 +0000 UTC m=+1551.269510639" watchObservedRunningTime="2026-10-30 19:30:03.236766719 +0000 UTC m=+1551.272279173"
Oct 30 19:30:03.310773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:03.310548    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" podStartSLOduration=3.310529701 podStartE2EDuration="3.310529701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.30893689 +0000 UTC m=+1551.344449356" watchObservedRunningTime="2026-10-30 19:30:03.310529701 +0000 UTC m=+1551.346042170"
Oct 30 19:30:04.206044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.206003    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerStarted","Data":"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"}
Oct 30 19:30:04.210481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.210446    4320 generic.go:356] "Generic (PLEG): container finished" podID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerID="4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d" exitCode=60
Oct 30 19:30:04.210747 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.210534    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerDied","Data":"4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"}
Oct 30 19:30:04.210972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.210959    4320 scope.go:122] "RemoveContainer" containerID="df15b53218ab6f538baddd07b29976c07972eb5e402756f03d27bfc0052446bb"
Oct 30 19:30:04.211914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.211835    4320 scope.go:122] "RemoveContainer" containerID="4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"
Oct 30 19:30:04.212259 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:04.212236    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7_odh-ai-gateway-infra(ef07e9d3-7122-4c01-8dd6-3585c9629f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06"
Oct 30 19:30:04.221153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.220088    4320 generic.go:356] "Generic (PLEG): container finished" podID="91840ea4-7c00-400d-b306-93c009cd3634" containerID="e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51" exitCode=60
Oct 30 19:30:04.221153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.220211    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerDied","Data":"e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"}
Oct 30 19:30:04.221153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.220667    4320 scope.go:122] "RemoveContainer" containerID="e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"
Oct 30 19:30:04.221153 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:04.221051    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb_odh-ai-gateway-infra(91840ea4-7c00-400d-b306-93c009cd3634)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" podUID="91840ea4-7c00-400d-b306-93c009cd3634"
Oct 30 19:30:04.231003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.226406    4320 generic.go:356] "Generic (PLEG): container finished" podID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerID="163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6" exitCode=60
Oct 30 19:30:04.231003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.226506    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerDied","Data":"163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"}
Oct 30 19:30:04.231003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.226983    4320 scope.go:122] "RemoveContainer" containerID="163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"
Oct 30 19:30:04.231003 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:04.227354    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx_odh-ai-gateway-infra(182c397d-fe6c-42d8-8d81-21d35144bdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba"
Oct 30 19:30:04.231003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.230799    4320 scope.go:122] "RemoveContainer" containerID="4a8f4632bdfcd6ad2d90a224f45679da17491d1e66f748a5828a625c74ad1e74"
Oct 30 19:30:04.231409 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:04.231177    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8_odh-ai-gateway-infra(82615a5a-a4f4-4e2f-9167-34e2ef758664)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664"
Oct 30 19:30:04.358184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.357765    4320 scope.go:122] "RemoveContainer" containerID="d0d4b715d581e4bb7fac1f4b73e317400c4abcbc43f1d60de3b64a5958ee6126"
Oct 30 19:30:04.454634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.452714    4320 scope.go:122] "RemoveContainer" containerID="2fe2ac921c13a99e8544a8c32f0d73b3c662bbe6c9c2e54a7817889e2c18a96e"
Oct 30 19:30:04.619579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.619510    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:30:04.686630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.682746    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:30:04.687766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.687735    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:30:04.688031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:04.687986    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:30:05.252129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:05.252097    4320 scope.go:122] "RemoveContainer" containerID="e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"
Oct 30 19:30:05.252580 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:05.252404    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb_odh-ai-gateway-infra(91840ea4-7c00-400d-b306-93c009cd3634)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" podUID="91840ea4-7c00-400d-b306-93c009cd3634"
Oct 30 19:30:05.254062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:05.254039    4320 scope.go:122] "RemoveContainer" containerID="163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"
Oct 30 19:30:05.254299 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:05.254276    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx_odh-ai-gateway-infra(182c397d-fe6c-42d8-8d81-21d35144bdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba"
Oct 30 19:30:05.256678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:05.256660    4320 scope.go:122] "RemoveContainer" containerID="4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"
Oct 30 19:30:05.256912 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:05.256893    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7_odh-ai-gateway-infra(ef07e9d3-7122-4c01-8dd6-3585c9629f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06"
Oct 30 19:30:05.678622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:05.677583    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:30:05.678622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:05.677659    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:30:06.686969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:06.686929    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:30:06.687536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:06.687505    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:30:07.347685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.347651    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:07.703819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.701294    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:30:07.703819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.701352    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:30:07.858641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.857667    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:07.977660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.968543    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\" (UniqueName: \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\") pod \"82615a5a-a4f4-4e2f-9167-34e2ef758664\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") "
Oct 30 19:30:07.977660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.968634    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\") pod \"82615a5a-a4f4-4e2f-9167-34e2ef758664\" (UID: \"82615a5a-a4f4-4e2f-9167-34e2ef758664\") "
Oct 30 19:30:07.981828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:07.981794    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca" pod "82615a5a-a4f4-4e2f-9167-34e2ef758664" (UID: "82615a5a-a4f4-4e2f-9167-34e2ef758664"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:08.003654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.003579    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv" pod "82615a5a-a4f4-4e2f-9167-34e2ef758664" (UID: "82615a5a-a4f4-4e2f-9167-34e2ef758664"). InnerVolumeSpecName "kube-api-access-92vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:08.006618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.006091    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82615a5a-a4f4-4e2f-9167-34e2ef758664-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:08.006792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.006766    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92vjv\" (UniqueName: \"kubernetes.io/projected/82615a5a-a4f4-4e2f-9167-34e2ef758664-kube-api-access-92vjv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:08.326871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.326785    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"
Oct 30 19:30:08.327053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.326783    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8" event={"ID":"82615a5a-a4f4-4e2f-9167-34e2ef758664","Type":"ContainerDied","Data":"e8ac1dca7d856b2bfddd4c779a515bd18ef5ef0238daacff4e8bf94b8a750114"}
Oct 30 19:30:08.327053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.326937    4320 scope.go:122] "RemoveContainer" containerID="4a8f4632bdfcd6ad2d90a224f45679da17491d1e66f748a5828a625c74ad1e74"
Oct 30 19:30:08.340887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.340863    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"]
Oct 30 19:30:08.344865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.344846    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5e08e43-zv548"]
Oct 30 19:30:08.396874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.396840    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:08.400833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.400809    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c5e08e43-29889810-hxbr8"]
Oct 30 19:30:08.689620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.683954    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:30:08.689620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:08.684019    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:30:09.620199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:09.620154    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" podUID="36cd6042-1a5a-4c21-adff-279ee94e76d3" containerName="payload-processing" containerID="cri-o://c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684" gracePeriod=25
Oct 30 19:30:09.919538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:09.919515    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:30:10.010721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.010678    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:30:10.010962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.010931    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" containerID="cri-o://4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425" gracePeriod=30
Oct 30 19:30:10.049475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.049429    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\" (UniqueName: \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\") pod \"36cd6042-1a5a-4c21-adff-279ee94e76d3\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") "
Oct 30 19:30:10.049712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.049636    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\") pod \"36cd6042-1a5a-4c21-adff-279ee94e76d3\" (UID: \"36cd6042-1a5a-4c21-adff-279ee94e76d3\") "
Oct 30 19:30:10.050991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.050948    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume" pod "36cd6042-1a5a-4c21-adff-279ee94e76d3" (UID: "36cd6042-1a5a-4c21-adff-279ee94e76d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:10.058469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.058434    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw" pod "36cd6042-1a5a-4c21-adff-279ee94e76d3" (UID: "36cd6042-1a5a-4c21-adff-279ee94e76d3"). InnerVolumeSpecName "kube-api-access-9fzqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:10.103765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.103672    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" path="/var/lib/kubelet/pods/82615a5a-a4f4-4e2f-9167-34e2ef758664/volumes"
Oct 30 19:30:10.110323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.104730    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6491d49-0e1a-4720-8f0a-a852ca811834" path="/var/lib/kubelet/pods/c6491d49-0e1a-4720-8f0a-a852ca811834/volumes"
Oct 30 19:30:10.165701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.164722    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzqw\" (UniqueName: \"kubernetes.io/projected/36cd6042-1a5a-4c21-adff-279ee94e76d3-kube-api-access-9fzqw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:10.165701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.164768    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd6042-1a5a-4c21-adff-279ee94e76d3-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:10.176623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.174546    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:30:10.345877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.345843    4320 generic.go:356] "Generic (PLEG): container finished" podID="36cd6042-1a5a-4c21-adff-279ee94e76d3" containerID="c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684" exitCode=0
Oct 30 19:30:10.346075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.345888    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"
Oct 30 19:30:10.346075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.345925    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" event={"ID":"36cd6042-1a5a-4c21-adff-279ee94e76d3","Type":"ContainerDied","Data":"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"}
Oct 30 19:30:10.346075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.345975    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl" event={"ID":"36cd6042-1a5a-4c21-adff-279ee94e76d3","Type":"ContainerDied","Data":"50ceab5ba7c9e1056d8e8001edd0703a60ea9eed3714a5419ddc30f1e504a095"}
Oct 30 19:30:10.346075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.345999    4320 scope.go:122] "RemoveContainer" containerID="c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"
Oct 30 19:30:10.356770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.356746    4320 scope.go:122] "RemoveContainer" containerID="c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"
Oct 30 19:30:10.357059 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:10.357039    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684\": container with ID starting with c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684 not found: ID does not exist" containerID="c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"
Oct 30 19:30:10.357121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.357067    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684"} err="failed to get container status \"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684\": rpc error: code = NotFound desc = could not find container \"c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684\": container with ID starting with c8d264161e86ef155d4c55bbd40acf861546ae18b2a832b20c4fab45697bb684 not found: ID does not exist"
Oct 30 19:30:10.407694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.407664    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:30:10.411772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.411744    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5e08e43-6b6bf6dd5b-rhwzl"]
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.718678    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.719776    4320 cpu_manager.go:414] "containerMap: removing container" podUID="36cd6042-1a5a-4c21-adff-279ee94e76d3" containerName="payload-processing"
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.719799    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cd6042-1a5a-4c21-adff-279ee94e76d3" containerName="payload-processing"
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.719832    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.719840    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.720038    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cd6042-1a5a-4c21-adff-279ee94e76d3" containerName="payload-processing"
Oct 30 19:30:10.722644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.720056    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:10.724704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.724683    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:10.739918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.739899    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-87b0dcce\""
Oct 30 19:30:10.740922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.740900    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-87b0dcce-dockercfg-m8gk2\""
Oct 30 19:30:10.741227 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.741207    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:30:10.882628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.881307    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fds\" (UniqueName: \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:10.882628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.881404    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:10.974977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.974891    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:30:10.975949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.975904    4320 patch_prober.go:29] interesting pod/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:30:10.976104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.975984    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:30:10.982574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.982545    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:10.982819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.982805    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fds\" (UniqueName: \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:10.983478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:10.983455    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:11.002910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:11.002885    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fds\" (UniqueName: \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\") pod \"payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:11.041458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:11.041425    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:11.461906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:11.461864    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:30:12.100642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.100586    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cd6042-1a5a-4c21-adff-279ee94e76d3" path="/var/lib/kubelet/pods/36cd6042-1a5a-4c21-adff-279ee94e76d3/volumes"
Oct 30 19:30:12.377308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.377225    4320 scope.go:122] "RemoveContainer" containerID="d224c2d5736d6f33f1d2d78383a88ca3fc4b3f99febd1b1e2337dc9e0f69cc02"
Oct 30 19:30:12.389362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.389338    4320 scope.go:122] "RemoveContainer" containerID="6afd8b5748ab5a78fc95e4a283a6c47ecf904c08fc11a41295cbc43533932dd3"
Oct 30 19:30:12.402023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.401998    4320 scope.go:122] "RemoveContainer" containerID="d7ae5b556510b7403515f123690dbdf1f8153d971681baa792a0cd99fd8983c6"
Oct 30 19:30:12.412492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.412471    4320 scope.go:122] "RemoveContainer" containerID="bb21743c6019f04a99482abf6df5c6926358ba903ea6a3bf8beacda729ead83e"
Oct 30 19:30:12.454024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.453997    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" event={"ID":"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46","Type":"ContainerStarted","Data":"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"}
Oct 30 19:30:12.454128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.454044    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" event={"ID":"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46","Type":"ContainerStarted","Data":"6f82c51869c918ff1e017002cb67067e335ee7a14100bec0beff752fe77dd5ef"}
Oct 30 19:30:12.454260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.454238    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:12.497039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.497012    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:12.497694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.497679    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:12.497743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.497694    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:12.497831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.497822    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82615a5a-a4f4-4e2f-9167-34e2ef758664" containerName="cleanup"
Oct 30 19:30:12.502235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.502217    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.505053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.505016    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" podStartSLOduration=2.505006682 podStartE2EDuration="2.505006682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:12.495840597 +0000 UTC m=+1560.531353062" watchObservedRunningTime="2026-10-30 19:30:12.505006682 +0000 UTC m=+1560.540519145"
Oct 30 19:30:12.518650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.517145    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f215f8b0-serving-cert\""
Oct 30 19:30:12.521154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.520956    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:12.608285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.608254    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.608464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.608299    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.608514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.608493    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwcs\" (UniqueName: \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.608561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.608520    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.709535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.709500    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwcs\" (UniqueName: \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.709535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.709547    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.709829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.709778    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.709866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.709827    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.710079 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:12.710056    4320 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-serving-cert: secret "maas-api-e2e-migrate-f215f8b0-serving-cert" not found
Oct 30 19:30:12.710160 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:12.710141    4320 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls podName:642cd2b3-eed8-4fe6-bb58-116dd44fd1a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:13.210119717 +0000 UTC m=+1561.245632174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls") pod "maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" (UID: "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5") : secret "maas-api-e2e-migrate-f215f8b0-serving-cert" not found
Oct 30 19:30:12.710384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.710357    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.713228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.713201    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:12.732033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:12.732001    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwcs\" (UniqueName: \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:13.216686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:13.216649    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:13.219113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:13.219092    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") pod \"maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:13.416567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:13.416527    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:13.758725 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:13.758680    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642cd2b3_eed8_4fe6_bb58_116dd44fd1a5.slice/crio-8b936f461ccb9d61da75aa589a7c205ba066c76fe4e715b8bcf5a637e15fa4fe WatchSource:0}: Error finding container 8b936f461ccb9d61da75aa589a7c205ba066c76fe4e715b8bcf5a637e15fa4fe: Status 404 returned error can't find the container with id 8b936f461ccb9d61da75aa589a7c205ba066c76fe4e715b8bcf5a637e15fa4fe
Oct 30 19:30:13.760191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:13.760157    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:13.760369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:13.760353    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:30:14.089960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:14.089880    4320 scope.go:122] "RemoveContainer" containerID="163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"
Oct 30 19:30:14.477154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:14.477119    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerStarted","Data":"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"}
Oct 30 19:30:14.477571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:14.477180    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:14.477571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:14.477197    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerStarted","Data":"8b936f461ccb9d61da75aa589a7c205ba066c76fe4e715b8bcf5a637e15fa4fe"}
Oct 30 19:30:14.533070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:14.533011    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" podStartSLOduration=2.082243213 podStartE2EDuration="2.532997s" totalImagesPullingTime="450.753787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:14.532578533 +0000 UTC m=+1562.568090999" watchObservedRunningTime="2026-10-30 19:30:14.532997 +0000 UTC m=+1562.568509467"
Oct 30 19:30:15.089032 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.089003    4320 scope.go:122] "RemoveContainer" containerID="4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"
Oct 30 19:30:15.408022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.408001    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:30:15.490483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.490444    4320 generic.go:356] "Generic (PLEG): container finished" podID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerID="5fab9b1486e6a9f044e7f66fb49e060ab38e8ba02096a1e7c8ba3948a0044a06" exitCode=60
Oct 30 19:30:15.490946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.490512    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerDied","Data":"5fab9b1486e6a9f044e7f66fb49e060ab38e8ba02096a1e7c8ba3948a0044a06"}
Oct 30 19:30:15.490946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.490576    4320 scope.go:122] "RemoveContainer" containerID="163f1b6039671da287ff07edbfb3f03cda2864aeade4582484c22aad04fe6ca6"
Oct 30 19:30:15.491082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.491001    4320 scope.go:122] "RemoveContainer" containerID="5fab9b1486e6a9f044e7f66fb49e060ab38e8ba02096a1e7c8ba3948a0044a06"
Oct 30 19:30:15.491399 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:15.491372    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx_odh-ai-gateway-infra(182c397d-fe6c-42d8-8d81-21d35144bdba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba"
Oct 30 19:30:15.493050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.493023    4320 generic.go:356] "Generic (PLEG): container finished" podID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerID="4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425" exitCode=0
Oct 30 19:30:15.493167 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.493082    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"
Oct 30 19:30:15.493167 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.493105    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerDied","Data":"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"}
Oct 30 19:30:15.493167 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.493157    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc" event={"ID":"8048a2cc-f686-47b8-adb2-8a96de69c666","Type":"ContainerDied","Data":"e155155a9abdadf1202e20bf2477a409e88f65dc91f21882967a1c63e023239a"}
Oct 30 19:30:15.495458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.495435    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerStarted","Data":"cf701cbdbfd6864ca65a8468599a0836827c954585e515bb350a26f1bdeb7514"}
Oct 30 19:30:15.504890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.504857    4320 scope.go:122] "RemoveContainer" containerID="4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"
Oct 30 19:30:15.516273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.516253    4320 scope.go:122] "RemoveContainer" containerID="7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"
Oct 30 19:30:15.526110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.526086    4320 scope.go:122] "RemoveContainer" containerID="4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"
Oct 30 19:30:15.526405 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:15.526370    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425\": container with ID starting with 4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425 not found: ID does not exist" containerID="4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"
Oct 30 19:30:15.526487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.526405    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425"} err="failed to get container status \"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425\": rpc error: code = NotFound desc = could not find container \"4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425\": container with ID starting with 4fe454d97a3294f75dc9c7b21597e408fc2c26b38858ee23b38325e85d86b425 not found: ID does not exist"
Oct 30 19:30:15.526487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.526431    4320 scope.go:122] "RemoveContainer" containerID="7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"
Oct 30 19:30:15.526725 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:15.526706    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de\": container with ID starting with 7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de not found: ID does not exist" containerID="7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"
Oct 30 19:30:15.526782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.526732    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de"} err="failed to get container status \"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de\": rpc error: code = NotFound desc = could not find container \"7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de\": container with ID starting with 7056e72c4214aa81e539e4630e9fbcdc5ff19ff3f701187704e1396644fb29de not found: ID does not exist"
Oct 30 19:30:15.565159 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565098    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565159 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565134    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565235    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565326    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:15.565380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565336    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565439    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565450    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:15.565488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565477    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565689    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565718    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:15.565824 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565783    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.565884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.565845    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\") pod \"8048a2cc-f686-47b8-adb2-8a96de69c666\" (UID: \"8048a2cc-f686-47b8-adb2-8a96de69c666\") "
Oct 30 19:30:15.566187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566156    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:15.566370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566343    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:15.566721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566696    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.566814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566726    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8048a2cc-f686-47b8-adb2-8a96de69c666-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.566814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566741    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.566814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566755    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.566814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.566767    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.567734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.567710    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:15.567837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.567822    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "kube-api-access-jtbw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:15.568124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.568099    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:15.568555 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.568539    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy" pod "8048a2cc-f686-47b8-adb2-8a96de69c666" (UID: "8048a2cc-f686-47b8-adb2-8a96de69c666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:15.668698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.668663    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.668829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.668743    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbw2\" (UniqueName: \"kubernetes.io/projected/8048a2cc-f686-47b8-adb2-8a96de69c666-kube-api-access-jtbw2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.668829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.668769    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.668829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.668784    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8048a2cc-f686-47b8-adb2-8a96de69c666-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.841633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.841544    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:30:15.853465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:15.853430    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c5e08e43-openshift-default-7fc9557759-w2hsc"]
Oct 30 19:30:16.100850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.100772    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" path="/var/lib/kubelet/pods/8048a2cc-f686-47b8-adb2-8a96de69c666/volumes"
Oct 30 19:30:16.440649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.440619    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:30:16.441293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.441277    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:16.441353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.441294    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:16.441353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.441316    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:16.441353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.441322    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:16.441448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.441424    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:16.445214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.445195    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.475042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.475019    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-249dd4-openshift-default-dockercfg-gbt9k\""
Oct 30 19:30:16.479544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.479518    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:30:16.506447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.506416    4320 generic.go:356] "Generic (PLEG): container finished" podID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerID="cf701cbdbfd6864ca65a8468599a0836827c954585e515bb350a26f1bdeb7514" exitCode=60
Oct 30 19:30:16.506914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.506491    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerDied","Data":"cf701cbdbfd6864ca65a8468599a0836827c954585e515bb350a26f1bdeb7514"}
Oct 30 19:30:16.506914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.506547    4320 scope.go:122] "RemoveContainer" containerID="4d539a50ce7240fcbf5a3d8cbd881558afa52506e46c377d147d5833fb75fe6d"
Oct 30 19:30:16.507048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.506980    4320 scope.go:122] "RemoveContainer" containerID="cf701cbdbfd6864ca65a8468599a0836827c954585e515bb350a26f1bdeb7514"
Oct 30 19:30:16.507318 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:16.507292    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7_odh-ai-gateway-infra(ef07e9d3-7122-4c01-8dd6-3585c9629f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06"
Oct 30 19:30:16.552776 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.552745    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:16.582502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582408    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.582502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582475    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.582773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582562    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.582773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582661    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.582861 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582784    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.582900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.582856    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.583065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.583046    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.583117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.583102    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.583156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.583126    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fw5\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.606335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.606309    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:16.684173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684140    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684191    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684218    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fw5\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684323    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684367    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684417    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684447    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684529    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684626    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684656    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.684997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.684968    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.685178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.685152    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.685275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.685254    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.685406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.685389    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.686768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.686745    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.686832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.686752    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.703657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.703636    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.703745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.703681    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fw5\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.731726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.731706    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:16.763150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.760733    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:16.790205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.789849    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\" (UniqueName: \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\") pod \"182c397d-fe6c-42d8-8d81-21d35144bdba\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") "
Oct 30 19:30:16.790361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.790253    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\") pod \"182c397d-fe6c-42d8-8d81-21d35144bdba\" (UID: \"182c397d-fe6c-42d8-8d81-21d35144bdba\") "
Oct 30 19:30:16.790952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.790924    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca" pod "182c397d-fe6c-42d8-8d81-21d35144bdba" (UID: "182c397d-fe6c-42d8-8d81-21d35144bdba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:16.792471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.792439    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6" pod "182c397d-fe6c-42d8-8d81-21d35144bdba" (UID: "182c397d-fe6c-42d8-8d81-21d35144bdba"). InnerVolumeSpecName "kube-api-access-2wbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:16.892499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.892461    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182c397d-fe6c-42d8-8d81-21d35144bdba-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:16.892499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:16.892494    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbm6\" (UniqueName: \"kubernetes.io/projected/182c397d-fe6c-42d8-8d81-21d35144bdba-kube-api-access-2wbm6\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:17.089459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.089432    4320 scope.go:122] "RemoveContainer" containerID="e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"
Oct 30 19:30:17.106274 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:17.106250    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9a8839_d5b5_4929_989a_4e6a19eb6ca9.slice/crio-2ad02b35f4abf60f05480119ffeab7a8ec7286736611d8f401613726779f145d WatchSource:0}: Error finding container 2ad02b35f4abf60f05480119ffeab7a8ec7286736611d8f401613726779f145d: Status 404 returned error can't find the container with id 2ad02b35f4abf60f05480119ffeab7a8ec7286736611d8f401613726779f145d
Oct 30 19:30:17.107230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.107204    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:30:17.108915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.108883    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:17.109000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.108946    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:17.109000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.108978    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:17.525430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.525392    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" event={"ID":"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9","Type":"ContainerStarted","Data":"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"}
Oct 30 19:30:17.527856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.527824    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" event={"ID":"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9","Type":"ContainerStarted","Data":"2ad02b35f4abf60f05480119ffeab7a8ec7286736611d8f401613726779f145d"}
Oct 30 19:30:17.529762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.529734    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx" event={"ID":"182c397d-fe6c-42d8-8d81-21d35144bdba","Type":"ContainerDied","Data":"d7c4708204290592ffaaefab8e95353468a8b52971b8021c15c59177c8a4a14a"}
Oct 30 19:30:17.529890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.529774    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"
Oct 30 19:30:17.529890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.529793    4320 scope.go:122] "RemoveContainer" containerID="5fab9b1486e6a9f044e7f66fb49e060ab38e8ba02096a1e7c8ba3948a0044a06"
Oct 30 19:30:17.627864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.627640    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:17.633254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.633226    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889810-8j8fx"]
Oct 30 19:30:17.680675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.679801    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podStartSLOduration=1.679783625 podStartE2EDuration="1.679783625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:17.677353472 +0000 UTC m=+1565.712865936" watchObservedRunningTime="2026-10-30 19:30:17.679783625 +0000 UTC m=+1565.715296092"
Oct 30 19:30:17.748948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.748919    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:17.761635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.761608    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:17.834303 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.834266    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\") pod \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") "
Oct 30 19:30:17.834449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.834325    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\" (UniqueName: \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\") pod \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\" (UID: \"ef07e9d3-7122-4c01-8dd6-3585c9629f06\") "
Oct 30 19:30:17.834762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.834735    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca" pod "ef07e9d3-7122-4c01-8dd6-3585c9629f06" (UID: "ef07e9d3-7122-4c01-8dd6-3585c9629f06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:17.835058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.835035    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef07e9d3-7122-4c01-8dd6-3585c9629f06-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:17.836983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.836956    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw" pod "ef07e9d3-7122-4c01-8dd6-3585c9629f06" (UID: "ef07e9d3-7122-4c01-8dd6-3585c9629f06"). InnerVolumeSpecName "kube-api-access-vb7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:17.937144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:17.937105    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7pw\" (UniqueName: \"kubernetes.io/projected/ef07e9d3-7122-4c01-8dd6-3585c9629f06-kube-api-access-vb7pw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:18.114143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.114109    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" path="/var/lib/kubelet/pods/182c397d-fe6c-42d8-8d81-21d35144bdba/volumes"
Oct 30 19:30:18.538808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.538709    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"
Oct 30 19:30:18.538808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.538736    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7" event={"ID":"ef07e9d3-7122-4c01-8dd6-3585c9629f06","Type":"ContainerDied","Data":"d80d90c65f4e5188ddb30bb1f3d7e827f857e0be4a2dd2579e28f4ec4f6d6893"}
Oct 30 19:30:18.538808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.538796    4320 scope.go:122] "RemoveContainer" containerID="cf701cbdbfd6864ca65a8468599a0836827c954585e515bb350a26f1bdeb7514"
Oct 30 19:30:18.541157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.541127    4320 generic.go:356] "Generic (PLEG): container finished" podID="91840ea4-7c00-400d-b306-93c009cd3634" containerID="352675dc4c23dd2a6233bb440ae4e3fa17619ff87454d5190dfae5e911828bb8" exitCode=60
Oct 30 19:30:18.541295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.541206    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerDied","Data":"352675dc4c23dd2a6233bb440ae4e3fa17619ff87454d5190dfae5e911828bb8"}
Oct 30 19:30:18.541818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.541782    4320 scope.go:122] "RemoveContainer" containerID="352675dc4c23dd2a6233bb440ae4e3fa17619ff87454d5190dfae5e911828bb8"
Oct 30 19:30:18.542098 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:18.542078    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb_odh-ai-gateway-infra(91840ea4-7c00-400d-b306-93c009cd3634)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" podUID="91840ea4-7c00-400d-b306-93c009cd3634"
Oct 30 19:30:18.549622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.549590    4320 scope.go:122] "RemoveContainer" containerID="e653978431a0f82d5b25e3c708127744bab3da5214006914dc987c909ee8ef51"
Oct 30 19:30:18.647655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.647623    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:18.651793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.651769    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889810-6nbp7"]
Oct 30 19:30:18.763232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.763203    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:18.763371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:18.763262    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:19.619469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.619434    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:19.762571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.761099    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:19.762571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.761172    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:19.854948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.854924    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:20.000715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.998256    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\") pod \"91840ea4-7c00-400d-b306-93c009cd3634\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") "
Oct 30 19:30:20.000715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:19.998323    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\" (UniqueName: \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\") pod \"91840ea4-7c00-400d-b306-93c009cd3634\" (UID: \"91840ea4-7c00-400d-b306-93c009cd3634\") "
Oct 30 19:30:20.000715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.000020    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca" pod "91840ea4-7c00-400d-b306-93c009cd3634" (UID: "91840ea4-7c00-400d-b306-93c009cd3634"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.005712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.005372    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp" pod "91840ea4-7c00-400d-b306-93c009cd3634" (UID: "91840ea4-7c00-400d-b306-93c009cd3634"). InnerVolumeSpecName "kube-api-access-kdvwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.117944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.109317    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91840ea4-7c00-400d-b306-93c009cd3634-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:20.117944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.109351    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvwp\" (UniqueName: \"kubernetes.io/projected/91840ea4-7c00-400d-b306-93c009cd3634-kube-api-access-kdvwp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:20.119208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.119177    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" path="/var/lib/kubelet/pods/ef07e9d3-7122-4c01-8dd6-3585c9629f06/volumes"
Oct 30 19:30:20.497442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.497390    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.104:8443/health\": dial tcp 10.133.0.104:8443: connect: connection refused"
Oct 30 19:30:20.560941 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.560917    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"
Oct 30 19:30:20.561125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.560914    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb" event={"ID":"91840ea4-7c00-400d-b306-93c009cd3634","Type":"ContainerDied","Data":"037fde4efdc04207d0fafeaf0772b132d6d567d8836227d9e3507ff7bb46c930"}
Oct 30 19:30:20.561125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.561093    4320 scope.go:122] "RemoveContainer" containerID="352675dc4c23dd2a6233bb440ae4e3fa17619ff87454d5190dfae5e911828bb8"
Oct 30 19:30:20.622889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.622854    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:20.626583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.626561    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6e0c55-29889810-dw2xb"]
Oct 30 19:30:20.761292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.761216    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:20.761292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:20.761266    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:21.761679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:21.761647    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:21.762042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:21.761704    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:22.103718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:22.103639    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91840ea4-7c00-400d-b306-93c009cd3634" path="/var/lib/kubelet/pods/91840ea4-7c00-400d-b306-93c009cd3634/volumes"
Oct 30 19:30:22.761532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:22.761494    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:22.761833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:22.761561    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:23.472036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:23.472007    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:30:23.761645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:23.761523    4320 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:30:23.761645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:23.761583    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:30:24.620845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.620756    4320 generic.go:356] "Generic (PLEG): container finished" podID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerID="97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b" exitCode=1
Oct 30 19:30:24.620845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.620831    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerDied","Data":"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"}
Oct 30 19:30:24.621309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.621293    4320 scope.go:122] "RemoveContainer" containerID="97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"
Oct 30 19:30:24.766006 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.765975    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:24.766406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.766383    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:24.767213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:24.767193    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.312342    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313331    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313352    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313376    4320 cpu_manager.go:414] "containerMap: removing container" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313384    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313405    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313413    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313451    4320 cpu_manager.go:414] "containerMap: removing container" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313458    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313479    4320 cpu_manager.go:414] "containerMap: removing container" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313486    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313501    4320 cpu_manager.go:414] "containerMap: removing container" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313509    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313681    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.313692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313694    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313708    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8048a2cc-f686-47b8-adb2-8a96de69c666" containerName="istio-proxy"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313746    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313758    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313767    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313778    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.317044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.313787    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:25.324910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.324885    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.348368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.348341    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a56cb5-openshift-default-dockercfg-2jc2r\""
Oct 30 19:30:25.351066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.350988    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 19:30:25.487981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.487944    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488095    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488211    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488308    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxfb\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488359    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488439    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488474    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488505    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.488655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.488538    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596641    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596747    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxfb\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596803    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596878    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596913    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596947    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.596979    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.597190    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.597256    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.597759    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.598670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.598003    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.599388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.598758    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.599388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.599323    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.623624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.599940    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.623624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.601989    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.641630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.633043    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.641630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.633276    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxfb\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.649616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.649074    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\") pod \"e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") " pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.661945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.661918    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:25.668550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.668256    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerStarted","Data":"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"}
Oct 30 19:30:25.669673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.669350    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:25.997306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:25.997282    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 19:30:25.998795 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:25.998760    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09713a4d_4542_4978_a9e8_9f2915794d50.slice/crio-96165eb459753dba46bd803cd4c90aa04d0ce259f4715d8323a235edecd418b3 WatchSource:0}: Error finding container 96165eb459753dba46bd803cd4c90aa04d0ce259f4715d8323a235edecd418b3: Status 404 returned error can't find the container with id 96165eb459753dba46bd803cd4c90aa04d0ce259f4715d8323a235edecd418b3
Oct 30 19:30:26.001562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.001529    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:26.001707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.001611    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:26.001707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.001644    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:26.680454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.680417    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerStarted","Data":"ec7d9966b1f335ec06da68d58e7ca0f17ca166ad613ccb31087eaf10b44d3c36"}
Oct 30 19:30:26.681027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.680993    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerStarted","Data":"96165eb459753dba46bd803cd4c90aa04d0ce259f4715d8323a235edecd418b3"}
Oct 30 19:30:26.738939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:26.738884    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podStartSLOduration=1.738869432 podStartE2EDuration="1.738869432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:26.735209413 +0000 UTC m=+1574.770721878" watchObservedRunningTime="2026-10-30 19:30:26.738869432 +0000 UTC m=+1574.774381895"
Oct 30 19:30:27.662908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:27.662876    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:27.669242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:27.669215    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:27.689716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:27.689690    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:27.690422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:27.690402    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:30:28.646646 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:28.646613    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:30:31.686077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:31.686051    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:32.842327 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:32.832075    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:30:34.569898 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.569866    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:30:34.570505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570490    4320 cpu_manager.go:414] "containerMap: removing container" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:34.570543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570506    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:34.570575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570545    4320 cpu_manager.go:414] "containerMap: removing container" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:34.570575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570550    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:34.570575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570570    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:34.570695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570576    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef07e9d3-7122-4c01-8dd6-3585c9629f06" containerName="cleanup"
Oct 30 19:30:34.570728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570696    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="91840ea4-7c00-400d-b306-93c009cd3634" containerName="cleanup"
Oct 30 19:30:34.570728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.570707    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="182c397d-fe6c-42d8-8d81-21d35144bdba" containerName="cleanup"
Oct 30 19:30:34.574284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.574264    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.596259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.596237    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:30:34.601284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.601260    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3bdb4fd2-serving-cert\""
Oct 30 19:30:34.652485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.652445    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.652655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.652576    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.652751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.652686    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2drz\" (UniqueName: \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.652825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.652799    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.754673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.754640    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2drz\" (UniqueName: \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.754857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.754757    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.754951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.754934    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.755033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.755016    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.755875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.755852    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.757489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.757467    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.757657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.757635    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.773527 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.773506    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2drz\" (UniqueName: \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\") pod \"maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:34.889231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:34.889202    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:35.214467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:35.214433    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:30:35.214690 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:35.214663    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8444a6_1ccf_4705_80e4_9f60ad28e62c.slice/crio-420c85114e041e7c3cf85b09b472a225cbcbf2fa62e3b66b9f44ab4635d9edaa WatchSource:0}: Error finding container 420c85114e041e7c3cf85b09b472a225cbcbf2fa62e3b66b9f44ab4635d9edaa: Status 404 returned error can't find the container with id 420c85114e041e7c3cf85b09b472a225cbcbf2fa62e3b66b9f44ab4635d9edaa
Oct 30 19:30:35.751235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:35.750431    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerStarted","Data":"bdc85f7ba188189c24df098044e909635b3366e12f6ed6b500efad5333d4b553"}
Oct 30 19:30:35.751235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:35.750503    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerStarted","Data":"420c85114e041e7c3cf85b09b472a225cbcbf2fa62e3b66b9f44ab4635d9edaa"}
Oct 30 19:30:35.751235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:35.750565    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:35.794116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:35.794071    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" podStartSLOduration=1.446839021 podStartE2EDuration="1.794053591s" totalImagesPullingTime="347.21457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.789875699 +0000 UTC m=+1583.825388164" watchObservedRunningTime="2026-10-30 19:30:35.794053591 +0000 UTC m=+1583.829566059"
Oct 30 19:30:41.760118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:41.760081    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": dial tcp 10.133.0.107:8443: connect: connection refused"
Oct 30 19:30:45.824955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:45.824923    4320 generic.go:356] "Generic (PLEG): container finished" podID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerID="bdc85f7ba188189c24df098044e909635b3366e12f6ed6b500efad5333d4b553" exitCode=1
Oct 30 19:30:45.825283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:45.824995    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerDied","Data":"bdc85f7ba188189c24df098044e909635b3366e12f6ed6b500efad5333d4b553"}
Oct 30 19:30:45.825455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:45.825435    4320 scope.go:122] "RemoveContainer" containerID="bdc85f7ba188189c24df098044e909635b3366e12f6ed6b500efad5333d4b553"
Oct 30 19:30:46.835241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:46.835205    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerStarted","Data":"069326cccb3f429e64671ebf1987746c0524a82d66f586b10f97a43a9a2b0950"}
Oct 30 19:30:46.835721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:46.835434    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:48.040663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:48.040628    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"]
Oct 30 19:30:48.051613 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:48.051576    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f138771-fnbd6"]
Oct 30 19:30:48.100836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:48.100801    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fbc46f-1d6f-4894-8158-c58270cc0b5a" path="/var/lib/kubelet/pods/02fbc46f-1d6f-4894-8158-c58270cc0b5a/volumes"
Oct 30 19:30:49.827937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.827904    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 19:30:49.833353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.833335    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.850915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.850890    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:30:49.857114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.857089    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 19:30:49.864665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.861937    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:30:49.864665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.861936    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:30:49.864665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.862042    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4lz7b\""
Oct 30 19:30:49.989383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989330    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrbx\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989397    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989437    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989589    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989683    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989739    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:49.989832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:49.989777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091313 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091215    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091313 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091285    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091328    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091514    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrbx\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091556    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.091677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091590    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.092337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.091753    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.092553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.092526    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.100646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.099880    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.100646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.100249    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.100646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.100469    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.100646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.100574    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.114114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.114079    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrbx\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.125485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.125370    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-54hm4\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.148050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.148023    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:50.514562 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:30:50.514534    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eae4b64_b77f_46b0_9e14_19b61a39b7d3.slice/crio-9df94400eb4b877c053023db9e51a69320114320540666dd607fefab000a838d WatchSource:0}: Error finding container 9df94400eb4b877c053023db9e51a69320114320540666dd607fefab000a838d: Status 404 returned error can't find the container with id 9df94400eb4b877c053023db9e51a69320114320540666dd607fefab000a838d
Oct 30 19:30:50.515371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.515344    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 19:30:50.882204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:50.882121    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" event={"ID":"8eae4b64-b77f-46b0-9e14-19b61a39b7d3","Type":"ContainerStarted","Data":"9df94400eb4b877c053023db9e51a69320114320540666dd607fefab000a838d"}
Oct 30 19:30:52.849119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:52.849090    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:30:53.385027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:53.384984    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:53.385158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:53.385056    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:53.912754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:53.912710    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" event={"ID":"8eae4b64-b77f-46b0-9e14-19b61a39b7d3","Type":"ContainerStarted","Data":"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"}
Oct 30 19:30:53.913205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:53.912986    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:53.985679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:53.985622    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" podStartSLOduration=2.119369187 podStartE2EDuration="4.985592972s" totalImagesPullingTime="2.866223785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:53.972300954 +0000 UTC m=+1602.007813416" watchObservedRunningTime="2026-10-30 19:30:53.985592972 +0000 UTC m=+1602.021105436"
Oct 30 19:30:54.923683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:54.923652    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 19:30:56.415672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.415639    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:56.416556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.416523    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api" containerID="cri-o://4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873" gracePeriod=30
Oct 30 19:30:56.878919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.878887    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:56.941401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.941362    4320 generic.go:356] "Generic (PLEG): container finished" podID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerID="4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873" exitCode=0
Oct 30 19:30:56.941573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.941423    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"
Oct 30 19:30:56.941573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.941439    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerDied","Data":"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"}
Oct 30 19:30:56.941573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.941504    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" event={"ID":"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5","Type":"ContainerDied","Data":"8b936f461ccb9d61da75aa589a7c205ba066c76fe4e715b8bcf5a637e15fa4fe"}
Oct 30 19:30:56.941573 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.941533    4320 scope.go:122] "RemoveContainer" containerID="4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"
Oct 30 19:30:56.945242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.945217    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") pod \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") "
Oct 30 19:30:56.945472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.945451    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\") pod \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") "
Oct 30 19:30:56.945570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.945549    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\" (UniqueName: \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\") pod \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") "
Oct 30 19:30:56.945695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.945632    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\") pod \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\" (UID: \"642cd2b3-eed8-4fe6-bb58-116dd44fd1a5\") "
Oct 30 19:30:56.946174 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.946030    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca" pod "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" (UID: "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:56.946418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.946394    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:56.948649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.948567    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs" pod "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" (UID: "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:56.948649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.948587    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls" pod "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" (UID: "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:56.959108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.959081    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs" pod "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" (UID: "642cd2b3-eed8-4fe6-bb58-116dd44fd1a5"). InnerVolumeSpecName "kube-api-access-zqwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.960416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.960385    4320 scope.go:122] "RemoveContainer" containerID="97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"
Oct 30 19:30:56.983230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.983196    4320 scope.go:122] "RemoveContainer" containerID="4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"
Oct 30 19:30:56.983539 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:56.983516    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873\": container with ID starting with 4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873 not found: ID does not exist" containerID="4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"
Oct 30 19:30:56.983646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.983554    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873"} err="failed to get container status \"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873\": rpc error: code = NotFound desc = could not find container \"4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873\": container with ID starting with 4fc5354ae3bb0630a8eddf17c5ab2c9354fa26e333424fcebf2dd41e3a6d3873 not found: ID does not exist"
Oct 30 19:30:56.983646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.983574    4320 scope.go:122] "RemoveContainer" containerID="97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"
Oct 30 19:30:56.983955 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:56.983916    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b\": container with ID starting with 97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b not found: ID does not exist" containerID="97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"
Oct 30 19:30:56.984008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:56.983964    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b"} err="failed to get container status \"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b\": rpc error: code = NotFound desc = could not find container \"97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b\": container with ID starting with 97288481643f4631cc7abe2b726cdfc7d9b4ecaadd930d5ea51a053f9a69bf8b not found: ID does not exist"
Oct 30 19:30:57.048467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:57.048436    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqwcs\" (UniqueName: \"kubernetes.io/projected/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-kube-api-access-zqwcs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:57.048467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:57.048468    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:57.048795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:57.048480    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:30:57.288683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:57.288656    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:57.292492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:57.292474    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5"]
Oct 30 19:30:58.099214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:58.099175    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" path="/var/lib/kubelet/pods/642cd2b3-eed8-4fe6-bb58-116dd44fd1a5/volumes"
Oct 30 19:30:58.645556 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:30:58.645519    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:30:59.742882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:30:59.742845    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f215f8b0-679dc7c859-7vbz5" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.104:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.473259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:12.473228    4320 scope.go:122] "RemoveContainer" containerID="958c730c937d6bf5f6402f2de4c5d8a7dd4111aff1c515a6e14177a2dd551c7f"
Oct 30 19:31:14.216821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:14.216758    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:31:18.188621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:18.187972    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"]
Oct 30 19:31:18.223620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:18.200353    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049f94ce-pchbl"]
Oct 30 19:31:19.084734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.084688    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:19.149392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.148694    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:31:19.208755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.208713    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:19.301092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.301050    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:31:19.310893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.310860    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:19.311393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:19.311079    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:20.105593 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:20.105557    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6719c0f4-adfc-4b10-85d7-b9b9d224d9b7" path="/var/lib/kubelet/pods/6719c0f4-adfc-4b10-85d7-b9b9d224d9b7/volumes"
Oct 30 19:31:26.488525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.488491    4320 generic.go:356] "Generic (PLEG): container finished" podID="09713a4d-4542-4978-a9e8-9f2915794d50" containerID="ec7d9966b1f335ec06da68d58e7ca0f17ca166ad613ccb31087eaf10b44d3c36" exitCode=0
Oct 30 19:31:26.488973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.488566    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerDied","Data":"ec7d9966b1f335ec06da68d58e7ca0f17ca166ad613ccb31087eaf10b44d3c36"}
Oct 30 19:31:26.489016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.488987    4320 scope.go:122] "RemoveContainer" containerID="ec7d9966b1f335ec06da68d58e7ca0f17ca166ad613ccb31087eaf10b44d3c36"
Oct 30 19:31:26.490431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.490400    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:26.490505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.490474    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:26.490543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:26.490505    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:27.503764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:27.503721    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerStarted","Data":"8b4c271fd4f3619100f9ae07b93e2c303a5087d928d21d1f370a3c76d317e6dc"}
Oct 30 19:31:27.662363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:27.662326    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:31:27.664394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:27.664361    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:31:27.664554 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:27.664437    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:31:27.699651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:27.693709    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:31:28.657376 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:31:28.657322    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:31:28.662947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:28.662914    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:31:28.663080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:28.662973    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:31:29.662219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:29.662184    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:31:29.662693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:29.662242    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:31:30.662448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:30.662404    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:31:30.662838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:30.662479    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:31:31.663134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:31.663103    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:31:31.663817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:31.663775    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:31:32.667062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:32.667034    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:31:32.668142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:32.668119    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 19:31:39.331124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.331085    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:31:39.331556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.331298    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api" containerID="cri-o://069326cccb3f429e64671ebf1987746c0524a82d66f586b10f97a43a9a2b0950" gracePeriod=30
Oct 30 19:31:39.738557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.738141    4320 generic.go:356] "Generic (PLEG): container finished" podID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerID="069326cccb3f429e64671ebf1987746c0524a82d66f586b10f97a43a9a2b0950" exitCode=0
Oct 30 19:31:39.738557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.738357    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerDied","Data":"069326cccb3f429e64671ebf1987746c0524a82d66f586b10f97a43a9a2b0950"}
Oct 30 19:31:39.738557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.738412    4320 scope.go:122] "RemoveContainer" containerID="bdc85f7ba188189c24df098044e909635b3366e12f6ed6b500efad5333d4b553"
Oct 30 19:31:39.818418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.818371    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:31:39.990869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.990767    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\" (UniqueName: \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\") pod \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") "
Oct 30 19:31:39.991041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.990888    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\") pod \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") "
Oct 30 19:31:39.991041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.990962    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\") pod \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") "
Oct 30 19:31:39.991175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.991140    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\") pod \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\" (UID: \"9d8444a6-1ccf-4705-80e4-9f60ad28e62c\") "
Oct 30 19:31:39.991466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.991432    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca" pod "9d8444a6-1ccf-4705-80e4-9f60ad28e62c" (UID: "9d8444a6-1ccf-4705-80e4-9f60ad28e62c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:39.991930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.991881    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:39.993723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.993698    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls" pod "9d8444a6-1ccf-4705-80e4-9f60ad28e62c" (UID: "9d8444a6-1ccf-4705-80e4-9f60ad28e62c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:39.993834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.993752    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs" pod "9d8444a6-1ccf-4705-80e4-9f60ad28e62c" (UID: "9d8444a6-1ccf-4705-80e4-9f60ad28e62c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:39.994468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:39.994446    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz" pod "9d8444a6-1ccf-4705-80e4-9f60ad28e62c" (UID: "9d8444a6-1ccf-4705-80e4-9f60ad28e62c"). InnerVolumeSpecName "kube-api-access-v2drz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:40.093547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.093512    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2drz\" (UniqueName: \"kubernetes.io/projected/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-kube-api-access-v2drz\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:40.093547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.093549    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:40.093916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.093562    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8444a6-1ccf-4705-80e4-9f60ad28e62c-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:40.759475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.755062    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd" event={"ID":"9d8444a6-1ccf-4705-80e4-9f60ad28e62c","Type":"ContainerDied","Data":"420c85114e041e7c3cf85b09b472a225cbcbf2fa62e3b66b9f44ab4635d9edaa"}
Oct 30 19:31:40.759475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.755130    4320 scope.go:122] "RemoveContainer" containerID="069326cccb3f429e64671ebf1987746c0524a82d66f586b10f97a43a9a2b0950"
Oct 30 19:31:40.759475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.755260    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"
Oct 30 19:31:40.855480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.855450    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:31:40.870087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:40.870022    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3bdb4fd2-fdddfd9fb-xxpjd"]
Oct 30 19:31:42.117621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:42.115371    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" path="/var/lib/kubelet/pods/9d8444a6-1ccf-4705-80e4-9f60ad28e62c/volumes"
Oct 30 19:31:43.048577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:43.048528    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:43.048971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:43.048920    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:53.029923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:53.029892    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"]
Oct 30 19:31:53.041529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:53.041502    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc73524f-8j5xx"]
Oct 30 19:31:54.098086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.098053    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa16902-d511-43d9-8ca6-665d101f87ad" path="/var/lib/kubelet/pods/0aa16902-d511-43d9-8ca6-665d101f87ad/volumes"
Oct 30 19:31:54.706860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.706829    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"]
Oct 30 19:31:54.707499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707485    4320 cpu_manager.go:414] "containerMap: removing container" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:31:54.707545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707500    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:31:54.707545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707534    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.707545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707539    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.707661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707550    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.707661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707555    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.707729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707672    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:31:54.707729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707685    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.707729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707692    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:31:54.707729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.707699    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8444a6-1ccf-4705-80e4-9f60ad28e62c" containerName="maas-api"
Oct 30 19:31:54.711009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.710990    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.729062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.729038    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"]
Oct 30 19:31:54.736348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.736321    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:54.736458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.736439    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:31:54.746380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.746346    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.746506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.746487    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh9n\" (UniqueName: \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.746641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.746612    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.847867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.847804    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.847980 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.847959    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.848109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.848092    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh9n\" (UniqueName: \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.848486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.848467    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.850260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.850239    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:54.867365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:54.867341    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh9n\" (UniqueName: \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\") pod \"maas-api-revoke-sub-a89d641d-zg59z\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:55.024513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:55.024455    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:55.325987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:55.325965    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"]
Oct 30 19:31:55.327056 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:31:55.327027    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381518d5_5806_4e61_9c0c_d13916d91b51.slice/crio-e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126 WatchSource:0}: Error finding container e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126: Status 404 returned error can't find the container with id e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126
Oct 30 19:31:55.899095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:55.899063    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z" event={"ID":"381518d5-5806-4e61-9c0c-d13916d91b51","Type":"ContainerStarted","Data":"ab2bbbd3c5374a7c5946210167473b5eb4a8c2354d2fb0680ba71f5792a79cfb"}
Oct 30 19:31:55.899283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:55.899136    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z" event={"ID":"381518d5-5806-4e61-9c0c-d13916d91b51","Type":"ContainerStarted","Data":"e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126"}
Oct 30 19:31:55.941531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:55.941480    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z" podStartSLOduration=1.941464511 podStartE2EDuration="1.941464511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:55.938560753 +0000 UTC m=+1663.974073224" watchObservedRunningTime="2026-10-30 19:31:55.941464511 +0000 UTC m=+1663.976976975"
Oct 30 19:31:56.918182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:56.918133    4320 generic.go:356] "Generic (PLEG): container finished" podID="381518d5-5806-4e61-9c0c-d13916d91b51" containerID="ab2bbbd3c5374a7c5946210167473b5eb4a8c2354d2fb0680ba71f5792a79cfb" exitCode=0
Oct 30 19:31:56.918182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:56.918172    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z" event={"ID":"381518d5-5806-4e61-9c0c-d13916d91b51","Type":"ContainerDied","Data":"ab2bbbd3c5374a7c5946210167473b5eb4a8c2354d2fb0680ba71f5792a79cfb"}
Oct 30 19:31:57.800709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:57.800673    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:31:58.154030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.154004    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:58.202375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.202340    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\") pod \"381518d5-5806-4e61-9c0c-d13916d91b51\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") "
Oct 30 19:31:58.202651 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.202621    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\" (UniqueName: \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\") pod \"381518d5-5806-4e61-9c0c-d13916d91b51\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") "
Oct 30 19:31:58.203094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.202765    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\") pod \"381518d5-5806-4e61-9c0c-d13916d91b51\" (UID: \"381518d5-5806-4e61-9c0c-d13916d91b51\") "
Oct 30 19:31:58.203248 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.203214    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca" pod "381518d5-5806-4e61-9c0c-d13916d91b51" (UID: "381518d5-5806-4e61-9c0c-d13916d91b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:58.203566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.203542    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:58.206080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.206058    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca" pod "381518d5-5806-4e61-9c0c-d13916d91b51" (UID: "381518d5-5806-4e61-9c0c-d13916d91b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:58.206230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.206213    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n" pod "381518d5-5806-4e61-9c0c-d13916d91b51" (UID: "381518d5-5806-4e61-9c0c-d13916d91b51"). InnerVolumeSpecName "kube-api-access-lrh9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:58.304982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.304959    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrh9n\" (UniqueName: \"kubernetes.io/projected/381518d5-5806-4e61-9c0c-d13916d91b51-kube-api-access-lrh9n\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:58.304982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.304984    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381518d5-5806-4e61-9c0c-d13916d91b51-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:31:58.645460 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:31:58.645427    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:31:58.948481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.948395    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z" event={"ID":"381518d5-5806-4e61-9c0c-d13916d91b51","Type":"ContainerDied","Data":"e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126"}
Oct 30 19:31:58.948481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.948431    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"
Oct 30 19:31:58.948481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:31:58.948456    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5d2672cd10b9c9d407c2a21f9315b356262011770ea0374c3dbcc2926483126"
Oct 30 19:32:12.524293 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:12.524258    4320 scope.go:122] "RemoveContainer" containerID="251f32663af51a9068aee0b2ce48fedba7d675e9cbb445fdeddd362e8499e93a"
Oct 30 19:32:22.033055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:22.033022    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"]
Oct 30 19:32:22.045359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:22.045329    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fecc27e-bkkcf"]
Oct 30 19:32:22.097862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:22.097832    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b9e04e-17ca-45a8-8847-de06cd479747" path="/var/lib/kubelet/pods/90b9e04e-17ca-45a8-8847-de06cd479747/volumes"
Oct 30 19:32:24.919334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:24.918357    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:32:28.644673 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:32:28.644636    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:32:34.915435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.915404    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"]
Oct 30 19:32:34.916080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.916063    4320 cpu_manager.go:414] "containerMap: removing container" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:32:34.916123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.916081    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642cd2b3-eed8-4fe6-bb58-116dd44fd1a5" containerName="maas-api"
Oct 30 19:32:34.916123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.916112    4320 cpu_manager.go:414] "containerMap: removing container" podUID="381518d5-5806-4e61-9c0c-d13916d91b51" containerName="revoke-keys"
Oct 30 19:32:34.916123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.916117    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381518d5-5806-4e61-9c0c-d13916d91b51" containerName="revoke-keys"
Oct 30 19:32:34.916251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.916242    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="381518d5-5806-4e61-9c0c-d13916d91b51" containerName="revoke-keys"
Oct 30 19:32:34.919809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.919789    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:34.937942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.937916    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"]
Oct 30 19:32:34.944774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.944749    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:34.944967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.944809    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:32:34.995472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.995445    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:34.995621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.995592    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsq4d\" (UniqueName: \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:34.995681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:34.995672    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.097357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.097335    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsq4d\" (UniqueName: \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.097507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.097388    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.097557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.097522    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.098083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.098057    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.099895 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.099871    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.116745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.116712    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsq4d\" (UniqueName: \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\") pod \"maas-api-revoke-sub-12e5c2c3-th6gb\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.234694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.234637    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:35.537654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:35.537571    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"]
Oct 30 19:32:35.537961 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:32:35.537939    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c23e57_07fa_4911_9cdd_d3a62f8881b4.slice/crio-63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531 WatchSource:0}: Error finding container 63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531: Status 404 returned error can't find the container with id 63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531
Oct 30 19:32:36.400023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:36.399979    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb" event={"ID":"22c23e57-07fa-4911-9cdd-d3a62f8881b4","Type":"ContainerStarted","Data":"f7f1fa2347783d95983b823f7ac3cda0902fa3fd1e5c7e1add2c9054320d3205"}
Oct 30 19:32:36.400497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:36.400035    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb" event={"ID":"22c23e57-07fa-4911-9cdd-d3a62f8881b4","Type":"ContainerStarted","Data":"63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531"}
Oct 30 19:32:36.443711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:36.443645    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb" podStartSLOduration=2.443630103 podStartE2EDuration="2.443630103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:36.441423756 +0000 UTC m=+1704.476936222" watchObservedRunningTime="2026-10-30 19:32:36.443630103 +0000 UTC m=+1704.479142566"
Oct 30 19:32:37.411872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:37.411838    4320 generic.go:356] "Generic (PLEG): container finished" podID="22c23e57-07fa-4911-9cdd-d3a62f8881b4" containerID="f7f1fa2347783d95983b823f7ac3cda0902fa3fd1e5c7e1add2c9054320d3205" exitCode=0
Oct 30 19:32:37.412250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:37.411922    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb" event={"ID":"22c23e57-07fa-4911-9cdd-d3a62f8881b4","Type":"ContainerDied","Data":"f7f1fa2347783d95983b823f7ac3cda0902fa3fd1e5c7e1add2c9054320d3205"}
Oct 30 19:32:38.607211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.607186    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:38.754073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.754005    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\" (UniqueName: \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\") pod \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") "
Oct 30 19:32:38.754236 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.754091    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\") pod \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") "
Oct 30 19:32:38.754309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.754250    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\") pod \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\" (UID: \"22c23e57-07fa-4911-9cdd-d3a62f8881b4\") "
Oct 30 19:32:38.754489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.754465    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca" pod "22c23e57-07fa-4911-9cdd-d3a62f8881b4" (UID: "22c23e57-07fa-4911-9cdd-d3a62f8881b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:38.754902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.754884    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:38.756208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.756180    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d" pod "22c23e57-07fa-4911-9cdd-d3a62f8881b4" (UID: "22c23e57-07fa-4911-9cdd-d3a62f8881b4"). InnerVolumeSpecName "kube-api-access-xsq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:38.756313 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.756238    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca" pod "22c23e57-07fa-4911-9cdd-d3a62f8881b4" (UID: "22c23e57-07fa-4911-9cdd-d3a62f8881b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:38.856273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.856241    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c23e57-07fa-4911-9cdd-d3a62f8881b4-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:38.856273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:38.856267    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsq4d\" (UniqueName: \"kubernetes.io/projected/22c23e57-07fa-4911-9cdd-d3a62f8881b4-kube-api-access-xsq4d\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:39.451671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:39.448640    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb" event={"ID":"22c23e57-07fa-4911-9cdd-d3a62f8881b4","Type":"ContainerDied","Data":"63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531"}
Oct 30 19:32:39.451671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:39.448703    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d501574c96d16c74a9ea7eadd1e083cfa178788ae1fdde140658fc5d1f8531"
Oct 30 19:32:39.451671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:39.448782    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"
Oct 30 19:32:41.049749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:41.049711    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"]
Oct 30 19:32:41.081304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:41.081180    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc23c18d-g6q6m"]
Oct 30 19:32:42.099115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:42.099083    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7602b3b-7fe2-4aaf-819a-a8583f52ffa6" path="/var/lib/kubelet/pods/f7602b3b-7fe2-4aaf-819a-a8583f52ffa6/volumes"
Oct 30 19:32:43.076180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.076138    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"]
Oct 30 19:32:43.076915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.076893    4320 cpu_manager.go:414] "containerMap: removing container" podUID="22c23e57-07fa-4911-9cdd-d3a62f8881b4" containerName="revoke-keys"
Oct 30 19:32:43.076915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.076915    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c23e57-07fa-4911-9cdd-d3a62f8881b4" containerName="revoke-keys"
Oct 30 19:32:43.077097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.077084    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c23e57-07fa-4911-9cdd-d3a62f8881b4" containerName="revoke-keys"
Oct 30 19:32:43.080972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.080952    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.098860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.098825    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"]
Oct 30 19:32:43.105995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.105969    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:43.106405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.106201    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:32:43.236967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.236934    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.237145 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.237122    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.237215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.237169    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jmzj\" (UniqueName: \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.339098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.339011    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.339098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.339059    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jmzj\" (UniqueName: \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.339344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.339246    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.339897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.339877    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.341388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.341365    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.363151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.363129    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jmzj\" (UniqueName: \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\") pod \"maas-api-revoke-sub-d7ce29ce-x4pcx\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.404021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.403997    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:43.710365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:43.710337    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"]
Oct 30 19:32:43.710745 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:32:43.710717    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45faa3ea_97fd_4e91_a462_9dc1169caba5.slice/crio-848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b WatchSource:0}: Error finding container 848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b: Status 404 returned error can't find the container with id 848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b
Oct 30 19:32:44.505951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:44.505914    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx" event={"ID":"45faa3ea-97fd-4e91-a462-9dc1169caba5","Type":"ContainerStarted","Data":"563487765ca06ef2e1d946351b4cf2f425ec0c0a6b8eb19f8bc50039f6d8dbcd"}
Oct 30 19:32:44.506696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:44.505983    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx" event={"ID":"45faa3ea-97fd-4e91-a462-9dc1169caba5","Type":"ContainerStarted","Data":"848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b"}
Oct 30 19:32:45.521618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:45.520860    4320 generic.go:356] "Generic (PLEG): container finished" podID="45faa3ea-97fd-4e91-a462-9dc1169caba5" containerID="563487765ca06ef2e1d946351b4cf2f425ec0c0a6b8eb19f8bc50039f6d8dbcd" exitCode=0
Oct 30 19:32:45.521618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:45.521082    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx" event={"ID":"45faa3ea-97fd-4e91-a462-9dc1169caba5","Type":"ContainerDied","Data":"563487765ca06ef2e1d946351b4cf2f425ec0c0a6b8eb19f8bc50039f6d8dbcd"}
Oct 30 19:32:45.979621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:45.949758    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 19:32:46.021765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.020822    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"]
Oct 30 19:32:46.026476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.026448    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.176273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.175434    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d9eb76-dockercfg-sttl2\""
Oct 30 19:32:46.176273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.176201    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d9eb76\""
Oct 30 19:32:46.278273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.225379    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.329335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.329295    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkpf\" (UniqueName: \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.343620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.338848    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 19:32:46.347618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.343835    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.433712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.433680    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.434032 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.434005    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.434242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.434222    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkpf\" (UniqueName: \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.434540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.434519    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llrq\" (UniqueName: \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.434798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.434777    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.435903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.435881    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.529949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.529871    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"]
Oct 30 19:32:46.536310 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.536279    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llrq\" (UniqueName: \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.577950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.577915    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.654312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.654288    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.655405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.646243    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkpf\" (UniqueName: \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\") pod \"payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:46.742678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.590140    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.742876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.669662    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llrq\" (UniqueName: \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.742974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.725969    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.830693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.824764    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:46.939755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:46.932933    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:47.108756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.108656    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"]
Oct 30 19:32:47.176697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.171757    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9049d375-6grgn"]
Oct 30 19:32:47.527843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.527813    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:47.674628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.670548    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx" event={"ID":"45faa3ea-97fd-4e91-a462-9dc1169caba5","Type":"ContainerDied","Data":"848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b"}
Oct 30 19:32:47.674628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.670677    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848a00337d2489536fcbaa5e24ac11c0c841eef6421c6a6487b2994c7b3a8b8b"
Oct 30 19:32:47.674628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.670764    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"
Oct 30 19:32:47.727667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.714489    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\" (UniqueName: \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\") pod \"45faa3ea-97fd-4e91-a462-9dc1169caba5\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") "
Oct 30 19:32:47.727667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.714610    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\") pod \"45faa3ea-97fd-4e91-a462-9dc1169caba5\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") "
Oct 30 19:32:47.727667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.714652    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\") pod \"45faa3ea-97fd-4e91-a462-9dc1169caba5\" (UID: \"45faa3ea-97fd-4e91-a462-9dc1169caba5\") "
Oct 30 19:32:47.745713 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.741579    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca" pod "45faa3ea-97fd-4e91-a462-9dc1169caba5" (UID: "45faa3ea-97fd-4e91-a462-9dc1169caba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:47.751429 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.747774    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj" pod "45faa3ea-97fd-4e91-a462-9dc1169caba5" (UID: "45faa3ea-97fd-4e91-a462-9dc1169caba5"). InnerVolumeSpecName "kube-api-access-7jmzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:47.781629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.779491    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca" pod "45faa3ea-97fd-4e91-a462-9dc1169caba5" (UID: "45faa3ea-97fd-4e91-a462-9dc1169caba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:47.844997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.844963    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jmzj\" (UniqueName: \"kubernetes.io/projected/45faa3ea-97fd-4e91-a462-9dc1169caba5-kube-api-access-7jmzj\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:47.844997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.844989    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:47.844997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.845005    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45faa3ea-97fd-4e91-a462-9dc1169caba5-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:47.856482 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:32:47.856452    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3b8e00_771d_4caa_b333_9c31a58b89d1.slice/crio-11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96 WatchSource:0}: Error finding container 11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96: Status 404 returned error can't find the container with id 11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96
Oct 30 19:32:47.856672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.856510    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"]
Oct 30 19:32:47.921107 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:32:47.921071    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06ceee5_3a98_4928_ac9f_e7e9cadeebca.slice/crio-c2873305ec2b3ae89763d58f96b4b64820b2d67759d9915a1eac518118eddfd0 WatchSource:0}: Error finding container c2873305ec2b3ae89763d58f96b4b64820b2d67759d9915a1eac518118eddfd0: Status 404 returned error can't find the container with id c2873305ec2b3ae89763d58f96b4b64820b2d67759d9915a1eac518118eddfd0
Oct 30 19:32:47.922329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:47.922308    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 19:32:48.099038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.099012    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7883575b-ab2d-4b62-a330-7e9c018078df" path="/var/lib/kubelet/pods/7883575b-ab2d-4b62-a330-7e9c018078df/volumes"
Oct 30 19:32:48.679375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.679342    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" event={"ID":"b06ceee5-3a98-4928-ac9f-e7e9cadeebca","Type":"ContainerStarted","Data":"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"}
Oct 30 19:32:48.679815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.679411    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:32:48.679815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.679439    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" event={"ID":"b06ceee5-3a98-4928-ac9f-e7e9cadeebca","Type":"ContainerStarted","Data":"c2873305ec2b3ae89763d58f96b4b64820b2d67759d9915a1eac518118eddfd0"}
Oct 30 19:32:48.680993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.680964    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf" event={"ID":"5a3b8e00-771d-4caa-b333-9c31a58b89d1","Type":"ContainerStarted","Data":"33eef0ff8498bb728144d90c00767abf1f0f70202be0cb36f0c84faef4ee6642"}
Oct 30 19:32:48.681104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.681013    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf" event={"ID":"5a3b8e00-771d-4caa-b333-9c31a58b89d1","Type":"ContainerStarted","Data":"11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96"}
Oct 30 19:32:48.721522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.721478    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" podStartSLOduration=3.721466915 podStartE2EDuration="3.721466915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:48.719159292 +0000 UTC m=+1716.754671757" watchObservedRunningTime="2026-10-30 19:32:48.721466915 +0000 UTC m=+1716.756979380"
Oct 30 19:32:48.761943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:48.761902    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf" podStartSLOduration=3.761889848 podStartE2EDuration="3.761889848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:48.759684317 +0000 UTC m=+1716.795196782" watchObservedRunningTime="2026-10-30 19:32:48.761889848 +0000 UTC m=+1716.797402312"
Oct 30 19:32:49.244336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.244301    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 19:32:49.245081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.245060    4320 cpu_manager.go:414] "containerMap: removing container" podUID="45faa3ea-97fd-4e91-a462-9dc1169caba5" containerName="revoke-keys"
Oct 30 19:32:49.245081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.245078    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45faa3ea-97fd-4e91-a462-9dc1169caba5" containerName="revoke-keys"
Oct 30 19:32:49.245268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.245233    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="45faa3ea-97fd-4e91-a462-9dc1169caba5" containerName="revoke-keys"
Oct 30 19:32:49.249510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.249489    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.269323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.269069    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.269323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.269166    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhw2z\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.269323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.269247    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.269567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.269527    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cd4bde-openshift-default-dockercfg-dhs49\""
Oct 30 19:32:49.270560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.269586    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.270736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.270671    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.270825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.270793    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.270884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.270859    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.270943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.270907    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.271008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.270963    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.283932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.283899    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372438    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372538    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhw2z\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372592    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372668    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372784    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372840    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372889    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372926    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.372989    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.373422    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.373514    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.373731    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.374405    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.375620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.375235    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.376917 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.376891    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.377588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.377566    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.397974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.397908    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhw2z\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.421217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.421131    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.569233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.569142    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:49.698151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.698108    4320 generic.go:356] "Generic (PLEG): container finished" podID="5a3b8e00-771d-4caa-b333-9c31a58b89d1" containerID="33eef0ff8498bb728144d90c00767abf1f0f70202be0cb36f0c84faef4ee6642" exitCode=0
Oct 30 19:32:49.699435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.699403    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf" event={"ID":"5a3b8e00-771d-4caa-b333-9c31a58b89d1","Type":"ContainerDied","Data":"33eef0ff8498bb728144d90c00767abf1f0f70202be0cb36f0c84faef4ee6642"}
Oct 30 19:32:49.933935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.933892    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 19:32:49.951411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.951365    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:49.951521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.951452    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:49.951521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:49.951498    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:50.707558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:50.707518    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerStarted","Data":"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"}
Oct 30 19:32:50.708050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:50.707584    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerStarted","Data":"132b219c8693ca774387d36c1eca493efff36363a30bf32a1e3d590dbcd254e9"}
Oct 30 19:32:50.779111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:50.778743    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podStartSLOduration=1.7787235209999999 podStartE2EDuration="1.778723521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:50.759632839 +0000 UTC m=+1718.795145306" watchObservedRunningTime="2026-10-30 19:32:50.778723521 +0000 UTC m=+1718.814235986"
Oct 30 19:32:50.949306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:50.948881    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:51.028790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.028688    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\") pod \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") "
Oct 30 19:32:51.028984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.028960    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\" (UniqueName: \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\") pod \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") "
Oct 30 19:32:51.029266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.029068    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\") pod \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\" (UID: \"5a3b8e00-771d-4caa-b333-9c31a58b89d1\") "
Oct 30 19:32:51.029532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.029501    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca" pod "5a3b8e00-771d-4caa-b333-9c31a58b89d1" (UID: "5a3b8e00-771d-4caa-b333-9c31a58b89d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:51.029957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.029915    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:51.038689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.038335    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca" pod "5a3b8e00-771d-4caa-b333-9c31a58b89d1" (UID: "5a3b8e00-771d-4caa-b333-9c31a58b89d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:51.041221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.041189    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq" pod "5a3b8e00-771d-4caa-b333-9c31a58b89d1" (UID: "5a3b8e00-771d-4caa-b333-9c31a58b89d1"). InnerVolumeSpecName "kube-api-access-2llrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:51.131525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.131498    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llrq\" (UniqueName: \"kubernetes.io/projected/5a3b8e00-771d-4caa-b333-9c31a58b89d1-kube-api-access-2llrq\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:51.131525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.131524    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a3b8e00-771d-4caa-b333-9c31a58b89d1-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:32:51.570219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.570189    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:51.575976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.575954    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:51.716024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.715994    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"
Oct 30 19:32:51.716387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.716027    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf" event={"ID":"5a3b8e00-771d-4caa-b333-9c31a58b89d1","Type":"ContainerDied","Data":"11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96"}
Oct 30 19:32:51.716387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.716062    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c9131d35f688264275823728e9733c91262f32f7626da4faa57caf7e34db96"
Oct 30 19:32:51.716387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.716312    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:51.717367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:51.717345    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:32:55.045104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.045072    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 19:32:55.045967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.045938    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5a3b8e00-771d-4caa-b333-9c31a58b89d1" containerName="revoke-keys"
Oct 30 19:32:55.045967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.045964    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3b8e00-771d-4caa-b333-9c31a58b89d1" containerName="revoke-keys"
Oct 30 19:32:55.046131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.046107    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3b8e00-771d-4caa-b333-9c31a58b89d1" containerName="revoke-keys"
Oct 30 19:32:55.050349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.050328    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.066963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.066940    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 19:32:55.082332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.082307    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cd4bde-dockercfg-cp7vc\""
Oct 30 19:32:55.082332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.082318    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cd4bde\""
Oct 30 19:32:55.307736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.302218    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-942hh\" (UniqueName: \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.307736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.305855    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.407692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.407649    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.407862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.407795    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-942hh\" (UniqueName: \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.408461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.408440    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.428443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.428411    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-942hh\" (UniqueName: \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\") pod \"payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.665750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.665720    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:55.971150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:55.971114    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 19:32:55.971390 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:32:55.971364    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7621967_ab48_4781_bda0_660f356dc505.slice/crio-d739c27e39eb322d82e12903b880cbf057dbea4d35fcdaaf89b32ec2a535bdd8 WatchSource:0}: Error finding container d739c27e39eb322d82e12903b880cbf057dbea4d35fcdaaf89b32ec2a535bdd8: Status 404 returned error can't find the container with id d739c27e39eb322d82e12903b880cbf057dbea4d35fcdaaf89b32ec2a535bdd8
Oct 30 19:32:56.476533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:56.476498    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:32:56.769041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:56.768940    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" event={"ID":"c7621967-ab48-4781-bda0-660f356dc505","Type":"ContainerStarted","Data":"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"}
Oct 30 19:32:56.769041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:56.768994    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" event={"ID":"c7621967-ab48-4781-bda0-660f356dc505","Type":"ContainerStarted","Data":"d739c27e39eb322d82e12903b880cbf057dbea4d35fcdaaf89b32ec2a535bdd8"}
Oct 30 19:32:56.769248 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:56.769077    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:32:56.816612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:56.816550    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" podStartSLOduration=1.816533365 podStartE2EDuration="1.816533365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:56.811628655 +0000 UTC m=+1724.847141116" watchObservedRunningTime="2026-10-30 19:32:56.816533365 +0000 UTC m=+1724.852045833"
Oct 30 19:32:58.645544 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:32:58.645513    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:32:59.702689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:32:59.702649    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 19:33:01.477617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.477547    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" podUID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" containerName="payload-pre-processing" containerID="cri-o://9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc" gracePeriod=25
Oct 30 19:33:01.773284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.773256    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:33:01.816930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.816888    4320 generic.go:356] "Generic (PLEG): container finished" podID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" containerID="9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc" exitCode=0
Oct 30 19:33:01.817100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.816935    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" event={"ID":"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46","Type":"ContainerDied","Data":"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"}
Oct 30 19:33:01.817100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.816948    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"
Oct 30 19:33:01.817100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.816987    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66" event={"ID":"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46","Type":"ContainerDied","Data":"6f82c51869c918ff1e017002cb67067e335ee7a14100bec0beff752fe77dd5ef"}
Oct 30 19:33:01.817100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.817016    4320 scope.go:122] "RemoveContainer" containerID="9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"
Oct 30 19:33:01.817363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.817339    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\" (UniqueName: \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\") pod \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") "
Oct 30 19:33:01.817649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.817583    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\") pod \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\" (UID: \"67e1c97c-f6cb-4d82-90f9-59bfabb0ef46\") "
Oct 30 19:33:01.818247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.818216    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume" pod "67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" (UID: "67e1c97c-f6cb-4d82-90f9-59bfabb0ef46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:01.818539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.818502    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:33:01.819840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.819816    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds" pod "67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" (UID: "67e1c97c-f6cb-4d82-90f9-59bfabb0ef46"). InnerVolumeSpecName "kube-api-access-v2fds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:01.847070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.846896    4320 scope.go:122] "RemoveContainer" containerID="9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"
Oct 30 19:33:01.847976 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:33:01.847912    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc\": container with ID starting with 9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc not found: ID does not exist" containerID="9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"
Oct 30 19:33:01.848155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.847993    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc"} err="failed to get container status \"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc\": rpc error: code = NotFound desc = could not find container \"9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc\": container with ID starting with 9bb55c06e06a18ef75da944b2edeeeca5b716b077ecbc5a4cf886dd3b26257fc not found: ID does not exist"
Oct 30 19:33:01.920439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:01.920410    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fds\" (UniqueName: \"kubernetes.io/projected/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46-kube-api-access-v2fds\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:33:02.161786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:02.161756    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:33:02.177776 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:02.177742    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-87b0dcce-6dc659767-z4h66"]
Oct 30 19:33:04.028560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:04.028516    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"]
Oct 30 19:33:04.040423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:04.040398    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb10009-kp7zb"]
Oct 30 19:33:04.099744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:04.099718    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" path="/var/lib/kubelet/pods/67e1c97c-f6cb-4d82-90f9-59bfabb0ef46/volumes"
Oct 30 19:33:04.100194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:04.100180    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729bea98-1e30-4252-96d6-b484a139050d" path="/var/lib/kubelet/pods/729bea98-1e30-4252-96d6-b484a139050d/volumes"
Oct 30 19:33:07.780681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:07.780645    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 19:33:10.448616 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:10.445428    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:33:12.572621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:12.564088    4320 scope.go:122] "RemoveContainer" containerID="d5dfc4e4f2b508f761cf4eb217b298995d36be922c0556d1a681ebf7ce67301e"
Oct 30 19:33:12.619915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:12.619888    4320 scope.go:122] "RemoveContainer" containerID="056cb69aa464c880dd96984f4a0a13e9f2ea8c5b3f829d7024cc9bd3fdc5f1ce"
Oct 30 19:33:13.549902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.549852    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 19:33:13.551189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.551162    4320 cpu_manager.go:414] "containerMap: removing container" podUID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" containerName="payload-pre-processing"
Oct 30 19:33:13.551351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.551337    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" containerName="payload-pre-processing"
Oct 30 19:33:13.551684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.551666    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e1c97c-f6cb-4d82-90f9-59bfabb0ef46" containerName="payload-pre-processing"
Oct 30 19:33:13.588082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.587358    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.593146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.593119    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 19:33:13.605020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.604998    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d9eb76-serving-cert\""
Oct 30 19:33:13.658435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.658400    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mw2\" (UniqueName: \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.662618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.658849    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.662926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.662904    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.663088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.663070    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.794622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.791766    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.794622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.791908    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.794622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.791997    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.794622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.792115    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mw2\" (UniqueName: \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.794622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.794508    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.857680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.820629    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.881191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.881156    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.912419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.912383    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mw2\" (UniqueName: \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\") pod \"maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:13.960208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:13.959909    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:15.370201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:15.370155    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:33:15.373326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:15.373236    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 19:33:15.376763 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:15.376718    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:15.435816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:15.435769    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:15.461258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:15.461214    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:16.263623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:16.263547    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" event={"ID":"414ca6a0-ed48-4c50-851d-8e2063ceed6a","Type":"ContainerStarted","Data":"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"}
Oct 30 19:33:16.263820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:16.263628    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" event={"ID":"414ca6a0-ed48-4c50-851d-8e2063ceed6a","Type":"ContainerStarted","Data":"ecddfd8d26b21bd27d8505e2cd83f5d6303459731134dc545cb892bdbe7a6e0a"}
Oct 30 19:33:16.263820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:16.263662    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:16.318570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:16.318500    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" podStartSLOduration=2.874045896 podStartE2EDuration="3.318480756s" totalImagesPullingTime="444.43486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:16.311244775 +0000 UTC m=+1744.346757241" watchObservedRunningTime="2026-10-30 19:33:16.318480756 +0000 UTC m=+1744.353993221"
Oct 30 19:33:18.711072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:18.711031    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"]
Oct 30 19:33:18.721593 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:18.721568    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-87b0dcce-wlszf"]
Oct 30 19:33:19.036840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:19.036756    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"]
Oct 30 19:33:19.048318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:19.048289    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44027af7-plgt4"]
Oct 30 19:33:20.099469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:20.099437    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3b8e00-771d-4caa-b333-9c31a58b89d1" path="/var/lib/kubelet/pods/5a3b8e00-771d-4caa-b333-9c31a58b89d1/volumes"
Oct 30 19:33:20.099950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:20.099933    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ed1050-20a5-44fc-ac64-2660961e0ee9" path="/var/lib/kubelet/pods/f7ed1050-20a5-44fc-ac64-2660961e0ee9/volumes"
Oct 30 19:33:22.277083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:22.277056    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 19:33:28.645678 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:33:28.645639    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:33:39.716024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.715985    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:33:39.720717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.720693    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.738016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.737987    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d9eb76-openshift-default-dockercfg-6vnqr\""
Oct 30 19:33:39.758314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.758277    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:33:39.815302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815260    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815350    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815386    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815457    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815493    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815514    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcp8q\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815563    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815639    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.815769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.815703    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916530    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916571    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916637    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916669    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916690    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcp8q\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.916745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916707    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916830    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.916920    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917193    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917264    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917424    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917428    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917664    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.917739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.917578    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.919263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.919238    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.919456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.919440    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.936163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.936140    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:39.936314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:39.936292    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcp8q\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:40.042531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.042465    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:40.369606 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.369563    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:33:40.370124 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:33:40.370101    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6882265c_10fe_49e8_ba44_9696912dd5a0.slice/crio-774f958aad9a52a51cd153913cafd7fa615930b66abc6e110caba8f90e2be5ce WatchSource:0}: Error finding container 774f958aad9a52a51cd153913cafd7fa615930b66abc6e110caba8f90e2be5ce: Status 404 returned error can't find the container with id 774f958aad9a52a51cd153913cafd7fa615930b66abc6e110caba8f90e2be5ce
Oct 30 19:33:40.372727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.372685    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:40.372826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.372759    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:40.372826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.372790    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:40.501419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.501380    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" event={"ID":"6882265c-10fe-49e8-ba44-9696912dd5a0","Type":"ContainerStarted","Data":"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"}
Oct 30 19:33:40.501759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.501730    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" event={"ID":"6882265c-10fe-49e8-ba44-9696912dd5a0","Type":"ContainerStarted","Data":"774f958aad9a52a51cd153913cafd7fa615930b66abc6e110caba8f90e2be5ce"}
Oct 30 19:33:40.548408 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:40.548355    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podStartSLOduration=1.548342316 podStartE2EDuration="1.548342316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:40.544732654 +0000 UTC m=+1768.580245103" watchObservedRunningTime="2026-10-30 19:33:40.548342316 +0000 UTC m=+1768.583854780"
Oct 30 19:33:41.042919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:41.042881    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:41.044370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:41.044346    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:41.044451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:41.044408    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:42.043872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:42.043840    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:42.044251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:42.043893    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:43.043306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:43.043273    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:43.043484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:43.043324    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:43.047753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:43.047725    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"]
Oct 30 19:33:43.059342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:43.059318    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b56bd5d8-xgzf8"]
Oct 30 19:33:44.043332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:44.043296    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:44.043521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:44.043349    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:44.118583 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:44.118548    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7206a119-8d87-4cd4-b7d0-43f8284f8180" path="/var/lib/kubelet/pods/7206a119-8d87-4cd4-b7d0-43f8284f8180/volumes"
Oct 30 19:33:45.042962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:45.042930    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:45.043138 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:45.042981    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:46.043000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:46.042967    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:33:46.043399 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:46.043025    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:33:47.047912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:47.047881    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:47.048390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:47.048283    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:47.049000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:47.048979    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:33:50.040468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:50.040432    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"]
Oct 30 19:33:50.044574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:50.044543    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e147182-lwmb9"]
Oct 30 19:33:50.098896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:50.098872    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b238ee1-c6b9-45da-b912-2d803abad2d9" path="/var/lib/kubelet/pods/2b238ee1-c6b9-45da-b912-2d803abad2d9/volumes"
Oct 30 19:33:54.006195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:33:54.006162    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:33:58.646480 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:33:58.646439    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:34:06.718207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.718174    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:06.718646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.718245    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:06.792921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.792889    4320 generic.go:356] "Generic (PLEG): container finished" podID="e9353dd8-420e-469b-9c85-429d115dd184" containerID="10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d" exitCode=0
Oct 30 19:34:06.793087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.792961    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerDied","Data":"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"}
Oct 30 19:34:06.793388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.793370    4320 scope.go:122] "RemoveContainer" containerID="10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"
Oct 30 19:34:06.795155 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.795126    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:06.795231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.795194    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:06.795274 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:06.795230    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:07.803199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:07.803164    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerStarted","Data":"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"}
Oct 30 19:34:08.099677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:08.089677    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"]
Oct 30 19:34:08.132190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:08.132156    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bf930a-qrsp6"]
Oct 30 19:34:08.607622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:08.605566    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:34:08.607622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:08.605688    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:08.607622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:08.605730    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:09.569995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.569959    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:09.570634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.570581    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:09.828831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.828718    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 19:34:09.832837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.832813    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.905070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.905029    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 19:34:09.948971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.948923    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.948992    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74bv\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949264    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949313    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949535    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949662    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949723    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949847    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:09.949976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:09.949959    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051274    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74bv\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051404    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051433    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051527    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051576    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051593    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051675    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051731    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.051831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.051771    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.052081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.052059    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.052372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.052352    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.052876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.052851    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.053047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.053022    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.053302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.053281    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.054928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.054909    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.055242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.055221    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.113479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.113391    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74bv\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.120285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.116066    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.129010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.128978    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b2a10f-9078-4cb1-8de3-987d44848151" path="/var/lib/kubelet/pods/13b2a10f-9078-4cb1-8de3-987d44848151/volumes"
Oct 30 19:34:10.147548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.147525    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:10.583877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.583316    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:10.583877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.583375    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:10.838692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.838201    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 19:34:10.848403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.848361    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:10.848647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.848623    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:10.848811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:10.848790    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:11.576883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:11.576845    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:11.577185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:11.577147    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:11.867348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:11.867245    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" event={"ID":"409bc3b1-677f-4e5e-ac36-f460e734c49c","Type":"ContainerStarted","Data":"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"}
Oct 30 19:34:11.867348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:11.867335    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" event={"ID":"409bc3b1-677f-4e5e-ac36-f460e734c49c","Type":"ContainerStarted","Data":"b6181fdac6fb7f753547249444ee69140551fbe23f5a375684b6713b39d3ea5f"}
Oct 30 19:34:12.149158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.149131    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:12.151101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.151064    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:12.151309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.151282    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:12.578674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.572861    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 19:34:12.578674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.572918    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 19:34:12.738096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.738053    4320 scope.go:122] "RemoveContainer" containerID="768d70694e14482ab6b27269c60246cae016274de03ca8b5a5fef4ff61607d49"
Oct 30 19:34:12.748673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.748653    4320 scope.go:122] "RemoveContainer" containerID="bf8f7f91e4744c32edf1fcee91389db5217938c1352eb2ba96611636eb909545"
Oct 30 19:34:12.757984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.757965    4320 scope.go:122] "RemoveContainer" containerID="85c55a0f0a28a04794f20a5dc632a4991a25c6f2a7ec3cf9e6d3736d048d36dd"
Oct 30 19:34:12.820978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:12.820951    4320 scope.go:122] "RemoveContainer" containerID="cc5f5d7c8c4fd3e6f2dde1b453c779ba4617e8b7b623afe9dbd8095defa45737"
Oct 30 19:34:13.148231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.148195    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:13.148755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.148258    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:13.596320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.594890    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:34:13.596320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.596033    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:34:13.597289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.597266    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 19:34:13.655708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:13.655661    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podStartSLOduration=4.655647343 podStartE2EDuration="4.655647343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:11.934961616 +0000 UTC m=+1799.970474083" watchObservedRunningTime="2026-10-30 19:34:13.655647343 +0000 UTC m=+1801.691159816"
Oct 30 19:34:14.147856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:14.147826    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:14.148012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:14.147873    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:15.163885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:15.159910    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:15.163885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:15.159975    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:16.148839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:16.148807    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:16.149002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:16.148860    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:16.900201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:16.900163    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:34:16.900672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:16.900416    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy" containerID="cri-o://5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb" gracePeriod=30
Oct 30 19:34:17.148073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:17.148038    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:34:17.148243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:17.148095    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 19:34:18.153997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:18.153966    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:18.154424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:18.154381    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:18.155018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:18.154997    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 19:34:22.191036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.191009    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:34:22.205520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205492    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205535    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205570    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205720    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205820    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205857    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:22.205947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205864    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.205947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205903    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:22.206109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.205947    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:22.206109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206056    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.206109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206070    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:22.206241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206123    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.206241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206200    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\") pod \"61456446-5707-4a7d-8f0d-bc34fe251d44\" (UID: \"61456446-5707-4a7d-8f0d-bc34fe251d44\") "
Oct 30 19:34:22.206442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206404    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:22.206959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206936    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.207071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206964    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.207071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.206982    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.207071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.207000    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.207071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.207014    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61456446-5707-4a7d-8f0d-bc34fe251d44-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.208008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.207983    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:22.208087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.208008    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:22.208232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.208213    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "kube-api-access-xp7xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:22.208284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.208238    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy" pod "61456446-5707-4a7d-8f0d-bc34fe251d44" (UID: "61456446-5707-4a7d-8f0d-bc34fe251d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:22.309259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.309225    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp7xf\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-kube-api-access-xp7xf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.309259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.309258    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.309439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.309272    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.309439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.309285    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61456446-5707-4a7d-8f0d-bc34fe251d44-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:22.994053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.994022    4320 generic.go:356] "Generic (PLEG): container finished" podID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerID="5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb" exitCode=0
Oct 30 19:34:22.994213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.994074    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"
Oct 30 19:34:22.994213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.994083    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" event={"ID":"61456446-5707-4a7d-8f0d-bc34fe251d44","Type":"ContainerDied","Data":"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"}
Oct 30 19:34:22.994213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.994123    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5" event={"ID":"61456446-5707-4a7d-8f0d-bc34fe251d44","Type":"ContainerDied","Data":"460842d646562bfdff98cde4313da0a9d1725a4971b1422b23197696ee5c37be"}
Oct 30 19:34:22.994213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:22.994143    4320 scope.go:122] "RemoveContainer" containerID="5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"
Oct 30 19:34:23.008763 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:23.008711    4320 scope.go:122] "RemoveContainer" containerID="5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"
Oct 30 19:34:23.009021 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:34:23.009001    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb\": container with ID starting with 5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb not found: ID does not exist" containerID="5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"
Oct 30 19:34:23.009068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:23.009030    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb"} err="failed to get container status \"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb\": rpc error: code = NotFound desc = could not find container \"5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb\": container with ID starting with 5e02cbfb87006c47d6e4f4aa46d4851ff50675ceefb6f9e89b1b530d6a094feb not found: ID does not exist"
Oct 30 19:34:23.060581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:23.060548    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:34:23.073240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:23.073214    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-md8k5"]
Oct 30 19:34:24.097896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.097862    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" path="/var/lib/kubelet/pods/61456446-5707-4a7d-8f0d-bc34fe251d44/volumes"
Oct 30 19:34:24.912953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.912924    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:34:24.913670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.913648    4320 cpu_manager.go:414] "containerMap: removing container" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy"
Oct 30 19:34:24.913770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.913671    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy"
Oct 30 19:34:24.913828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.913806    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="61456446-5707-4a7d-8f0d-bc34fe251d44" containerName="istio-proxy"
Oct 30 19:34:24.917225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.917203    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949548    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949654    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949732    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949799 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59hv8\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949845    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.949928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949908    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.950020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.949999    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.950119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.950107    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.950157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.950140    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:24.967009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:24.966975    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:34:25.052518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052479    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.052768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052638    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.052768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052688    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.052768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052745    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.052948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052824    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.052960    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053054    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59hv8\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053166    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053221    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053351    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053637    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053760 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053680    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.053760 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.053682    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.054430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.054399    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.058350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.058319    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.059774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.059752    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.059976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.059960    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"]
Oct 30 19:34:25.064875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.064856    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9fe01a-rpzcb"]
Oct 30 19:34:25.079394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.078421    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.084681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.084636    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59hv8\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.231115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.231020    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:25.551322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.551298    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:34:25.552925 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:34:25.552895    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e05e26_57fc_45cf_8ada_b1d41a642ed2.slice/crio-107fbb8ca6c3e42a321009f35288271ffb862810811c393f2235bd2b1f8e48dd WatchSource:0}: Error finding container 107fbb8ca6c3e42a321009f35288271ffb862810811c393f2235bd2b1f8e48dd: Status 404 returned error can't find the container with id 107fbb8ca6c3e42a321009f35288271ffb862810811c393f2235bd2b1f8e48dd
Oct 30 19:34:25.555976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.555941    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:25.556057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.556012    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:25.556057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:25.556041    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:26.034741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.034706    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" event={"ID":"80e05e26-57fc-45cf-8ada-b1d41a642ed2","Type":"ContainerStarted","Data":"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"}
Oct 30 19:34:26.034915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.034763    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" event={"ID":"80e05e26-57fc-45cf-8ada-b1d41a642ed2","Type":"ContainerStarted","Data":"107fbb8ca6c3e42a321009f35288271ffb862810811c393f2235bd2b1f8e48dd"}
Oct 30 19:34:26.081244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.081185    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podStartSLOduration=2.081171974 podStartE2EDuration="2.081171974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:26.079752772 +0000 UTC m=+1814.115265250" watchObservedRunningTime="2026-10-30 19:34:26.081171974 +0000 UTC m=+1814.116684438"
Oct 30 19:34:26.098333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.098304    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b0fb16-8399-43a9-8876-4f50417a6c10" path="/var/lib/kubelet/pods/26b0fb16-8399-43a9-8876-4f50417a6c10/volumes"
Oct 30 19:34:26.231725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.231694    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:26.233400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.233371    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:26.233534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.233420    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:26.914625 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.911171    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"]
Oct 30 19:34:26.929802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.929774    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"]
Oct 30 19:34:26.930040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.929860    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:26.944668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.944647    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:26.944796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:26.944778    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:34:27.025204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.024625    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwfk\" (UniqueName: \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.025204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.024826    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.025204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.025160    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.126995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.126957    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.127153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.127078    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwfk\" (UniqueName: \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.127261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.127241    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.127700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.127681    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.129647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.129626    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.148198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.148176    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwfk\" (UniqueName: \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\") pod \"maas-api-revoke-sub-3ccc4e2b-mxzl7\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.232347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.232266    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:27.232347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.232322    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:27.248516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.248489    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:27.604948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:27.604861    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"]
Oct 30 19:34:27.609275 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:34:27.609249    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36c19bb_23c7_4740_9418_a894bd695753.slice/crio-4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9 WatchSource:0}: Error finding container 4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9: Status 404 returned error can't find the container with id 4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9
Oct 30 19:34:28.064132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:28.064093    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7" event={"ID":"f36c19bb-23c7-4740-9418-a894bd695753","Type":"ContainerStarted","Data":"4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9"}
Oct 30 19:34:28.232123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:28.232085    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:28.232283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:28.232140    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:28.646655 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:34:28.646614    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:34:29.096672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:29.096354    4320 generic.go:356] "Generic (PLEG): container finished" podID="f36c19bb-23c7-4740-9418-a894bd695753" containerID="c52ea0b0928940b9c8fd095308bf7516b2e2cbc229e29a6c9df78f0932ece3e0" exitCode=0
Oct 30 19:34:29.096672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:29.096511    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7" event={"ID":"f36c19bb-23c7-4740-9418-a894bd695753","Type":"ContainerDied","Data":"c52ea0b0928940b9c8fd095308bf7516b2e2cbc229e29a6c9df78f0932ece3e0"}
Oct 30 19:34:29.233240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:29.233201    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:29.233460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:29.233274    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:30.232056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.232018    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:30.232440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.232091    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:30.315451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.315420    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:30.442770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.442740    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\") pod \"f36c19bb-23c7-4740-9418-a894bd695753\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") "
Oct 30 19:34:30.443058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.443044    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\" (UniqueName: \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\") pod \"f36c19bb-23c7-4740-9418-a894bd695753\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") "
Oct 30 19:34:30.443157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.443147    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\") pod \"f36c19bb-23c7-4740-9418-a894bd695753\" (UID: \"f36c19bb-23c7-4740-9418-a894bd695753\") "
Oct 30 19:34:30.444259 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.444233    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca" pod "f36c19bb-23c7-4740-9418-a894bd695753" (UID: "f36c19bb-23c7-4740-9418-a894bd695753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:30.446928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.446897    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca" pod "f36c19bb-23c7-4740-9418-a894bd695753" (UID: "f36c19bb-23c7-4740-9418-a894bd695753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:30.447213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.447189    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk" pod "f36c19bb-23c7-4740-9418-a894bd695753" (UID: "f36c19bb-23c7-4740-9418-a894bd695753"). InnerVolumeSpecName "kube-api-access-nhwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:30.545011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.544973    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:30.545011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.545008    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwfk\" (UniqueName: \"kubernetes.io/projected/f36c19bb-23c7-4740-9418-a894bd695753-kube-api-access-nhwfk\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:30.545197 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:30.545022    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36c19bb-23c7-4740-9418-a894bd695753-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:31.127464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:31.127422    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7" event={"ID":"f36c19bb-23c7-4740-9418-a894bd695753","Type":"ContainerDied","Data":"4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9"}
Oct 30 19:34:31.127680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:31.127538    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4804f5d5c4ccb0d7f701c607f365adfbf85b035bebd97c8332abe0301d381fb9"
Oct 30 19:34:31.127680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:31.127642    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"
Oct 30 19:34:31.232301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:31.232260    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:34:31.232791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:31.232323    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:34:32.354624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:32.354570    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:32.355015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:32.354970    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:32.355846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:32.355821    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:34:49.880014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.879960    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"]
Oct 30 19:34:49.881166 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.881136    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f36c19bb-23c7-4740-9418-a894bd695753" containerName="revoke-keys"
Oct 30 19:34:49.881353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.881333    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36c19bb-23c7-4740-9418-a894bd695753" containerName="revoke-keys"
Oct 30 19:34:49.881755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.881724    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36c19bb-23c7-4740-9418-a894bd695753" containerName="revoke-keys"
Oct 30 19:34:49.889950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.889927    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:49.905683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.905648    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"]
Oct 30 19:34:49.917020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.916990    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:49.917163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:49.917050    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:34:50.007956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.007916    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.008143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.008106    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7b9w\" (UniqueName: \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.008349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.008326    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.110366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.110335    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.110569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.110422    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.110569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.110525    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7b9w\" (UniqueName: \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.111139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.111115    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.112987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.112961    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.130611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.130539    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7b9w\" (UniqueName: \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.219802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.219777    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:50.554491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:50.554448    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"]
Oct 30 19:34:51.456047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:51.456004    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m" event={"ID":"74aada7e-70cb-4573-b57d-f2c428f8a5bf","Type":"ContainerStarted","Data":"1bb01c5e66ae800e1c80b0719865fa8c7eb50800221deb98a7fa0ed5a8944514"}
Oct 30 19:34:51.456526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:51.456077    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m" event={"ID":"74aada7e-70cb-4573-b57d-f2c428f8a5bf","Type":"ContainerStarted","Data":"bda0ec0270a3de2506bcee12d5087e772ec7824b2f77bd1e734c289007e950e3"}
Oct 30 19:34:51.501331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:51.501280    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m" podStartSLOduration=2.501266747 podStartE2EDuration="2.501266747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:51.497167336 +0000 UTC m=+1839.532679800" watchObservedRunningTime="2026-10-30 19:34:51.501266747 +0000 UTC m=+1839.536779211"
Oct 30 19:34:52.465531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:52.465494    4320 generic.go:356] "Generic (PLEG): container finished" podID="74aada7e-70cb-4573-b57d-f2c428f8a5bf" containerID="1bb01c5e66ae800e1c80b0719865fa8c7eb50800221deb98a7fa0ed5a8944514" exitCode=0
Oct 30 19:34:52.466017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:52.465584    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m" event={"ID":"74aada7e-70cb-4573-b57d-f2c428f8a5bf","Type":"ContainerDied","Data":"1bb01c5e66ae800e1c80b0719865fa8c7eb50800221deb98a7fa0ed5a8944514"}
Oct 30 19:34:53.945381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:53.945356    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:54.025101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.025062    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\") pod \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") "
Oct 30 19:34:54.025279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.025259    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\") pod \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") "
Oct 30 19:34:54.025510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.025486    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\" (UniqueName: \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\") pod \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\" (UID: \"74aada7e-70cb-4573-b57d-f2c428f8a5bf\") "
Oct 30 19:34:54.025706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.025677    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca" pod "74aada7e-70cb-4573-b57d-f2c428f8a5bf" (UID: "74aada7e-70cb-4573-b57d-f2c428f8a5bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:54.026214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.026187    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:54.027849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.027820    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca" pod "74aada7e-70cb-4573-b57d-f2c428f8a5bf" (UID: "74aada7e-70cb-4573-b57d-f2c428f8a5bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:54.027936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.027911    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w" pod "74aada7e-70cb-4573-b57d-f2c428f8a5bf" (UID: "74aada7e-70cb-4573-b57d-f2c428f8a5bf"). InnerVolumeSpecName "kube-api-access-b7b9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:54.128215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.128187    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7b9w\" (UniqueName: \"kubernetes.io/projected/74aada7e-70cb-4573-b57d-f2c428f8a5bf-kube-api-access-b7b9w\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:54.128340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.128220    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aada7e-70cb-4573-b57d-f2c428f8a5bf-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:34:54.651950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.651925    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"
Oct 30 19:34:54.652122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.651964    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m" event={"ID":"74aada7e-70cb-4573-b57d-f2c428f8a5bf","Type":"ContainerDied","Data":"bda0ec0270a3de2506bcee12d5087e772ec7824b2f77bd1e734c289007e950e3"}
Oct 30 19:34:54.652122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:34:54.652012    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda0ec0270a3de2506bcee12d5087e772ec7824b2f77bd1e734c289007e950e3"
Oct 30 19:34:58.646438 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:34:58.646400    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:35:06.947623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:06.947416    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:35:12.869874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:12.869834    4320 scope.go:122] "RemoveContainer" containerID="9f556ee1cdc8d22337b087bf49dd7093cb576fa5e3b639ef875f22a471439e18"
Oct 30 19:35:12.882819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:12.882797    4320 scope.go:122] "RemoveContainer" containerID="091d36fb30e27cf25436867da8fcce3eaf6d44814af80c27f81826d772bd68a6"
Oct 30 19:35:12.893462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:12.893430    4320 scope.go:122] "RemoveContainer" containerID="1589d439f021e17fab54af115ab5e6b07079a29df8db7bb657e75998f48af0ba"
Oct 30 19:35:12.902739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:12.902721    4320 scope.go:122] "RemoveContainer" containerID="10c2cb2197204f70d37b51fa6723a4c9f8810f00d811ae5ccd1ee68c902f5a98"
Oct 30 19:35:14.711930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.711894    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"]
Oct 30 19:35:14.712607 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.712580    4320 cpu_manager.go:414] "containerMap: removing container" podUID="74aada7e-70cb-4573-b57d-f2c428f8a5bf" containerName="revoke-keys"
Oct 30 19:35:14.712664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.712613    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74aada7e-70cb-4573-b57d-f2c428f8a5bf" containerName="revoke-keys"
Oct 30 19:35:14.712819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.712806    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="74aada7e-70cb-4573-b57d-f2c428f8a5bf" containerName="revoke-keys"
Oct 30 19:35:14.718393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.718371    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.737188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.737158    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"]
Oct 30 19:35:14.740565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.740543    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:35:14.740733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.740683    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:14.835587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.835553    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.835741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.835640    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.835789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.835736    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q477t\" (UniqueName: \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.937260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.937230    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.937384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.937275    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.937384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.937315    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q477t\" (UniqueName: \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.937884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.937863    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.939503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.939484    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:14.956692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:14.956673    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q477t\" (UniqueName: \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\") pod \"maas-api-revoke-sub-822c0b74-5qb77\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:15.034010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:15.033950    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:15.351758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:15.350688    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"]
Oct 30 19:35:15.893885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:15.893852    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77" event={"ID":"ab87df39-991c-47cc-a2bd-ec65b76b2267","Type":"ContainerStarted","Data":"de148771c26792e2e9a0a609abac0990f50f227c90dac7a54cffa8f45a99a9e0"}
Oct 30 19:35:15.894246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:15.893901    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77" event={"ID":"ab87df39-991c-47cc-a2bd-ec65b76b2267","Type":"ContainerStarted","Data":"a474dee4d5a5ca1f7fcdc104e1ed9a4dd0aa3d8f6920ec4cba6f554c6882ff49"}
Oct 30 19:35:15.937638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:15.937536    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77" podStartSLOduration=1.937522924 podStartE2EDuration="1.937522924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:15.935984027 +0000 UTC m=+1863.971496492" watchObservedRunningTime="2026-10-30 19:35:15.937522924 +0000 UTC m=+1863.973035387"
Oct 30 19:35:16.904105 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:16.904067    4320 generic.go:356] "Generic (PLEG): container finished" podID="ab87df39-991c-47cc-a2bd-ec65b76b2267" containerID="de148771c26792e2e9a0a609abac0990f50f227c90dac7a54cffa8f45a99a9e0" exitCode=0
Oct 30 19:35:16.904498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:16.904161    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77" event={"ID":"ab87df39-991c-47cc-a2bd-ec65b76b2267","Type":"ContainerDied","Data":"de148771c26792e2e9a0a609abac0990f50f227c90dac7a54cffa8f45a99a9e0"}
Oct 30 19:35:17.434817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:17.434787    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:35:17.435010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:17.434972    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy" containerID="cri-o://69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793" gracePeriod=30
Oct 30 19:35:18.083835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.083808    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:18.099010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.098981    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\" (UniqueName: \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\") pod \"ab87df39-991c-47cc-a2bd-ec65b76b2267\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") "
Oct 30 19:35:18.099146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.099037    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\") pod \"ab87df39-991c-47cc-a2bd-ec65b76b2267\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") "
Oct 30 19:35:18.099351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.099328    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\") pod \"ab87df39-991c-47cc-a2bd-ec65b76b2267\" (UID: \"ab87df39-991c-47cc-a2bd-ec65b76b2267\") "
Oct 30 19:35:18.099703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.099680    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca" pod "ab87df39-991c-47cc-a2bd-ec65b76b2267" (UID: "ab87df39-991c-47cc-a2bd-ec65b76b2267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:18.100016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.099998    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:18.101056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.101033    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t" pod "ab87df39-991c-47cc-a2bd-ec65b76b2267" (UID: "ab87df39-991c-47cc-a2bd-ec65b76b2267"). InnerVolumeSpecName "kube-api-access-q477t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:18.101461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.101444    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca" pod "ab87df39-991c-47cc-a2bd-ec65b76b2267" (UID: "ab87df39-991c-47cc-a2bd-ec65b76b2267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:18.201361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.201298    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q477t\" (UniqueName: \"kubernetes.io/projected/ab87df39-991c-47cc-a2bd-ec65b76b2267-kube-api-access-q477t\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:18.201361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.201326    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab87df39-991c-47cc-a2bd-ec65b76b2267-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:18.921634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.921580    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"
Oct 30 19:35:18.921634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.921617    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77" event={"ID":"ab87df39-991c-47cc-a2bd-ec65b76b2267","Type":"ContainerDied","Data":"a474dee4d5a5ca1f7fcdc104e1ed9a4dd0aa3d8f6920ec4cba6f554c6882ff49"}
Oct 30 19:35:18.921841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:18.921675    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a474dee4d5a5ca1f7fcdc104e1ed9a4dd0aa3d8f6920ec4cba6f554c6882ff49"
Oct 30 19:35:22.729025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.729001    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:35:22.765798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.765772    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.765912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.765901    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.765983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.765925    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.765983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.765946    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.765983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.765971    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.766157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766008    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.766157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766084    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.766157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766131    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.766157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766136    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:22.766365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766191    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\") pod \"733aef98-88e7-41d7-92b3-ae12b8090371\" (UID: \"733aef98-88e7-41d7-92b3-ae12b8090371\") "
Oct 30 19:35:22.766510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766451    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:22.766592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766566    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:22.766772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.766740    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:22.767101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.767053    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:22.767205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.767131    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733aef98-88e7-41d7-92b3-ae12b8090371-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.767205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.767154    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.767205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.767173    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.767205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.767189    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.768384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.768359    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:22.768494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.768478    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "kube-api-access-vhn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:22.768636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.768581    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:22.768773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.768753    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token" pod "733aef98-88e7-41d7-92b3-ae12b8090371" (UID: "733aef98-88e7-41d7-92b3-ae12b8090371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:22.869107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.869081    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733aef98-88e7-41d7-92b3-ae12b8090371-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.869107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.869106    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.869285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.869118    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.869285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.869126    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhn2f\" (UniqueName: \"kubernetes.io/projected/733aef98-88e7-41d7-92b3-ae12b8090371-kube-api-access-vhn2f\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.869285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.869136    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733aef98-88e7-41d7-92b3-ae12b8090371-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:22.963443 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.963360    4320 generic.go:356] "Generic (PLEG): container finished" podID="733aef98-88e7-41d7-92b3-ae12b8090371" containerID="69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793" exitCode=0
Oct 30 19:35:22.963764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.963427    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"
Oct 30 19:35:22.963910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.963446    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" event={"ID":"733aef98-88e7-41d7-92b3-ae12b8090371","Type":"ContainerDied","Data":"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"}
Oct 30 19:35:22.963995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.963957    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m" event={"ID":"733aef98-88e7-41d7-92b3-ae12b8090371","Type":"ContainerDied","Data":"3e58503cbd48bc9eab1e2cef9cc1bb9f6c4dab2ddf6970cd23c71f21e6a7607a"}
Oct 30 19:35:22.964053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.964022    4320 scope.go:122] "RemoveContainer" containerID="69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"
Oct 30 19:35:22.983475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.983449    4320 scope.go:122] "RemoveContainer" containerID="69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"
Oct 30 19:35:22.983749 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:35:22.983715    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793\": container with ID starting with 69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793 not found: ID does not exist" containerID="69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"
Oct 30 19:35:22.983871 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:22.983749    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793"} err="failed to get container status \"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793\": rpc error: code = NotFound desc = could not find container \"69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793\": container with ID starting with 69a78985ca60eb7787fe2782b40471b2e86f2d9d6f44130b7a2b374ce9dfe793 not found: ID does not exist"
Oct 30 19:35:23.019982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:23.019959    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:35:23.023699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:23.023677    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-z2k6m"]
Oct 30 19:35:24.100712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:24.100675    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" path="/var/lib/kubelet/pods/733aef98-88e7-41d7-92b3-ae12b8090371/volumes"
Oct 30 19:35:27.209523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:27.209489    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:35:27.210012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:27.209728    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" containerID="cri-o://09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324" gracePeriod=30
Oct 30 19:35:28.649748 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:35:28.649700    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:35:32.226030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.225990    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:41934->10.133.0.48:15021: read: connection reset by peer" start-of-body=
Oct 30 19:35:32.226410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.226055    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:41934->10.133.0.48:15021: read: connection reset by peer"
Oct 30 19:35:32.500862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.500835    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:35:32.603332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603296    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.603489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603460    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.603610 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603573    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.603709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603673    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:32.603773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603713    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.603832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603813    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:32.603997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.603946    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.604136 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604048    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.604136 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604107    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.604272 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604201    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.604272 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604248    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\") pod \"4159106d-5f94-484b-b1dd-e843018750f6\" (UID: \"4159106d-5f94-484b-b1dd-e843018750f6\") "
Oct 30 19:35:32.604454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604412    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:32.604564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604547    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:32.604674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.604636    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:32.605366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.605199    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.605366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.605230    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.605366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.605244    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4159106d-5f94-484b-b1dd-e843018750f6-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.605366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.605256    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.605366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.605271    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.607306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.607277    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:32.607458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.607428    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:32.608753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.608720    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:32.608987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.608961    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8" pod "4159106d-5f94-484b-b1dd-e843018750f6" (UID: "4159106d-5f94-484b-b1dd-e843018750f6"). InnerVolumeSpecName "kube-api-access-dhzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:32.645950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.645914    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:35:32.646266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.646233    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy" containerID="cri-o://c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e" gracePeriod=30
Oct 30 19:35:32.707078 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.707044    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4159106d-5f94-484b-b1dd-e843018750f6-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.707224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.707083    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzr8\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-kube-api-access-dhzr8\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.707224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.707101    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4159106d-5f94-484b-b1dd-e843018750f6-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:32.707224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:32.707116    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4159106d-5f94-484b-b1dd-e843018750f6-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:33.100614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.100562    4320 generic.go:356] "Generic (PLEG): container finished" podID="4159106d-5f94-484b-b1dd-e843018750f6" containerID="09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324" exitCode=0
Oct 30 19:35:33.100791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.100627    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" event={"ID":"4159106d-5f94-484b-b1dd-e843018750f6","Type":"ContainerDied","Data":"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"}
Oct 30 19:35:33.100791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.100678    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r" event={"ID":"4159106d-5f94-484b-b1dd-e843018750f6","Type":"ContainerDied","Data":"6c028506aa53bd06d8f6d075ab457f7c540edf413fceeb1924f2413abee98925"}
Oct 30 19:35:33.100791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.100700    4320 scope.go:122] "RemoveContainer" containerID="09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"
Oct 30 19:35:33.100791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.100637    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"
Oct 30 19:35:33.112748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.112666    4320 scope.go:122] "RemoveContainer" containerID="09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"
Oct 30 19:35:33.112963 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:35:33.112945    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324\": container with ID starting with 09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324 not found: ID does not exist" containerID="09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"
Oct 30 19:35:33.113019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.112973    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324"} err="failed to get container status \"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324\": rpc error: code = NotFound desc = could not find container \"09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324\": container with ID starting with 09a4e39cbe9255f2933142945ea3ea853cde316b34b46be8434bd41147996324 not found: ID does not exist"
Oct 30 19:35:33.153729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.153705    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:35:33.165411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:33.165386    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-8w67r"]
Oct 30 19:35:34.099885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:34.099854    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4159106d-5f94-484b-b1dd-e843018750f6" path="/var/lib/kubelet/pods/4159106d-5f94-484b-b1dd-e843018750f6/volumes"
Oct 30 19:35:36.564661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.564584    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"]
Oct 30 19:35:36.565330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565312    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy"
Oct 30 19:35:36.565383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565332    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy"
Oct 30 19:35:36.565383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565350    4320 cpu_manager.go:414] "containerMap: removing container" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy"
Oct 30 19:35:36.565383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565355    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy"
Oct 30 19:35:36.565383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565366    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ab87df39-991c-47cc-a2bd-ec65b76b2267" containerName="revoke-keys"
Oct 30 19:35:36.565383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565371    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87df39-991c-47cc-a2bd-ec65b76b2267" containerName="revoke-keys"
Oct 30 19:35:36.565533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565483    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4159106d-5f94-484b-b1dd-e843018750f6" containerName="istio-proxy"
Oct 30 19:35:36.565533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565493    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="733aef98-88e7-41d7-92b3-ae12b8090371" containerName="istio-proxy"
Oct 30 19:35:36.565533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.565499    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87df39-991c-47cc-a2bd-ec65b76b2267" containerName="revoke-keys"
Oct 30 19:35:36.569268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.569244    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.590169 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.590142    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"]
Oct 30 19:35:36.593683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.593654    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:36.593805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.593682    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:35:36.668718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.668689    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.668919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.668899    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.668985 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.668971    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59th\" (UniqueName: \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.770089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.770057    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.770222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.770206    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.770287 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.770271    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59th\" (UniqueName: \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.770711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.770683    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.772633 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.772611    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.790076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.790048    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59th\" (UniqueName: \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\") pod \"maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:36.897987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:36.897965    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:37.200242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:37.200210    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"]
Oct 30 19:35:37.201246 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:35:37.201223    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2492d9f9_60e9_4ffc_b6ab_c53b88d3f672.slice/crio-501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa WatchSource:0}: Error finding container 501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa: Status 404 returned error can't find the container with id 501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa
Oct 30 19:35:37.973201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:37.973175    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:35:38.096796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.096771    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.096838    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.096913    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.096958    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097008    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097049    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097111    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097167    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.097415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097320    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\") pod \"3d29bea6-5523-4842-ab2d-6d719147ccf4\" (UID: \"3d29bea6-5523-4842-ab2d-6d719147ccf4\") "
Oct 30 19:35:38.098876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.097979    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:38.098876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.098335    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.098876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.098556    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:38.099388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.099366    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:38.103640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.100112    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:38.103640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.101086    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:38.103640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.101499    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:38.103640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.102521    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "kube-api-access-rzg48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:38.103640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.103104    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:38.103887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.103717    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert" pod "3d29bea6-5523-4842-ab2d-6d719147ccf4" (UID: "3d29bea6-5523-4842-ab2d-6d719147ccf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:38.140945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.140921    4320 generic.go:356] "Generic (PLEG): container finished" podID="2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" containerID="707ea20213cb8cea775b81bfdadf861d8a0282105b72b0e29202aa4038927f7b" exitCode=0
Oct 30 19:35:38.142528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.142508    4320 generic.go:356] "Generic (PLEG): container finished" podID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerID="c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e" exitCode=0
Oct 30 19:35:38.142645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.142579    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"
Oct 30 19:35:38.151547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.151525    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9" event={"ID":"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672","Type":"ContainerDied","Data":"707ea20213cb8cea775b81bfdadf861d8a0282105b72b0e29202aa4038927f7b"}
Oct 30 19:35:38.151701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.151555    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9" event={"ID":"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672","Type":"ContainerStarted","Data":"501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa"}
Oct 30 19:35:38.151701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.151570    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" event={"ID":"3d29bea6-5523-4842-ab2d-6d719147ccf4","Type":"ContainerDied","Data":"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"}
Oct 30 19:35:38.151701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.151591    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w" event={"ID":"3d29bea6-5523-4842-ab2d-6d719147ccf4","Type":"ContainerDied","Data":"cf5d70b2bf443ec5ad37f9206cd44768de2484458f39aac1269260653531d466"}
Oct 30 19:35:38.151701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.151625    4320 scope.go:122] "RemoveContainer" containerID="c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"
Oct 30 19:35:38.164208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.164031    4320 scope.go:122] "RemoveContainer" containerID="c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"
Oct 30 19:35:38.164405 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:35:38.164376    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e\": container with ID starting with c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e not found: ID does not exist" containerID="c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"
Oct 30 19:35:38.164458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.164415    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e"} err="failed to get container status \"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e\": rpc error: code = NotFound desc = could not find container \"c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e\": container with ID starting with c97eed4e4486d379095f8b57cbcda55b4288bafd4ed842b716f95b886798d71e not found: ID does not exist"
Oct 30 19:35:38.200266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200230    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200268    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200283    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200297    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzg48\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-kube-api-access-rzg48\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200311    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200326    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d29bea6-5523-4842-ab2d-6d719147ccf4-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200338    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.200499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.200351    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d29bea6-5523-4842-ab2d-6d719147ccf4-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:38.222885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.222855    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:35:38.228988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:38.228961    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-52z6w"]
Oct 30 19:35:39.324044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.324020    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:39.416011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.415937    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\") pod \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") "
Oct 30 19:35:39.416261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.416241    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\" (UniqueName: \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\") pod \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") "
Oct 30 19:35:39.416329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.416302    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\") pod \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\" (UID: \"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672\") "
Oct 30 19:35:39.416390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.416338    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca" pod "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" (UID: "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:39.417039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.417011    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:39.418968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.418926    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca" pod "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" (UID: "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:39.419072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.419004    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th" pod "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" (UID: "2492d9f9-60e9-4ffc-b6ab-c53b88d3f672"). InnerVolumeSpecName "kube-api-access-c59th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:39.518268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.518240    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c59th\" (UniqueName: \"kubernetes.io/projected/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-kube-api-access-c59th\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:39.518268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:39.518266    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:35:40.098322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:40.098291    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" path="/var/lib/kubelet/pods/3d29bea6-5523-4842-ab2d-6d719147ccf4/volumes"
Oct 30 19:35:40.160283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:40.160259    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"
Oct 30 19:35:40.160410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:40.160303    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9" event={"ID":"2492d9f9-60e9-4ffc-b6ab-c53b88d3f672","Type":"ContainerDied","Data":"501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa"}
Oct 30 19:35:40.160410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:40.160348    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501e2d4a7a899cafc3bcf93e473e8a63022af8bba86253ca6ba64d389f2ed9fa"
Oct 30 19:35:45.072521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:45.072467    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:35:55.765924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.765890    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:35:55.766617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766587    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" containerName="revoke-keys"
Oct 30 19:35:55.766737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766621    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" containerName="revoke-keys"
Oct 30 19:35:55.766737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766654    4320 cpu_manager.go:414] "containerMap: removing container" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy"
Oct 30 19:35:55.766737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766662    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy"
Oct 30 19:35:55.767113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766800    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" containerName="revoke-keys"
Oct 30 19:35:55.767113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.766811    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d29bea6-5523-4842-ab2d-6d719147ccf4" containerName="istio-proxy"
Oct 30 19:35:55.771889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.771869    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.814057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.801873    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-287ea2-openshift-default-dockercfg-7kw4v\""
Oct 30 19:35:55.814057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.809714    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:35:55.831297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831268    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831329    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831400    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831459    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrcb\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831497    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831532    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831638    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831681    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.831795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.831780    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933037    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933080    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933135    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933174    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrcb\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933279    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933389    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933426    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933492    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933535    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933829    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933876    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.933987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.933894    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.934223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.934205    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.935582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.935561    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.935741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.935720    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.953128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.953108    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrcb\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:55.953240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:55.953138    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:56.093490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:56.093392    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:56.415157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:56.415125    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:35:56.416943 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:35:56.416915    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b17461_7a83_4c4b_b51b_cc07a82c9aa5.slice/crio-d4d2f5a65f805aceb46dfa1bfa05f26e17b0f643ca2718ee4dbd5f2279b1db6c WatchSource:0}: Error finding container d4d2f5a65f805aceb46dfa1bfa05f26e17b0f643ca2718ee4dbd5f2279b1db6c: Status 404 returned error can't find the container with id d4d2f5a65f805aceb46dfa1bfa05f26e17b0f643ca2718ee4dbd5f2279b1db6c
Oct 30 19:35:56.420481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:56.420446    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:56.420579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:56.420529    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:56.420579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:56.420559    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:57.299764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:57.299733    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"]
Oct 30 19:35:57.313127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:57.313100    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-bp89m"]
Oct 30 19:35:57.362551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:57.362522    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" event={"ID":"73b17461-7a83-4c4b-b51b-cc07a82c9aa5","Type":"ContainerStarted","Data":"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"}
Oct 30 19:35:57.362758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:57.362576    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" event={"ID":"73b17461-7a83-4c4b-b51b-cc07a82c9aa5","Type":"ContainerStarted","Data":"d4d2f5a65f805aceb46dfa1bfa05f26e17b0f643ca2718ee4dbd5f2279b1db6c"}
Oct 30 19:35:57.405311 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:57.405253    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podStartSLOduration=2.405233696 podStartE2EDuration="2.405233696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:57.403911881 +0000 UTC m=+1905.439424346" watchObservedRunningTime="2026-10-30 19:35:57.405233696 +0000 UTC m=+1905.440746159"
Oct 30 19:35:58.098800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:58.098767    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:35:58.099010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:58.098835    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 19:35:58.105836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:58.105808    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74aada7e-70cb-4573-b57d-f2c428f8a5bf" path="/var/lib/kubelet/pods/74aada7e-70cb-4573-b57d-f2c428f8a5bf/volumes"
Oct 30 19:35:58.106158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:58.106145    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:35:58.649428 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:35:58.649387    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:35:59.094073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:59.093982    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:35:59.094073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:35:59.094036    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 19:36:00.093956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:00.093924    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:36:00.094380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:00.093984    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 19:36:01.094228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:01.094153    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:36:01.094792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:01.094227    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 19:36:02.097333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:02.097304    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:36:02.097785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:02.097685    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 19:36:03.104459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:03.104427    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:36:03.105626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:03.105588    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:36:03.106541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:03.106515    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:36:12.952994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:12.952960    4320 scope.go:122] "RemoveContainer" containerID="68cd6dbda11dbe6ff9d2da5b884fe65842ba30dd351a014f4df93557f64f3120"
Oct 30 19:36:12.963182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:12.963159    4320 scope.go:122] "RemoveContainer" containerID="1805183adb55985fceb5909133eea35509e92f80fb6a8d9dd8519da5b5c6fd45"
Oct 30 19:36:12.972705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:12.972684    4320 scope.go:122] "RemoveContainer" containerID="7736308a74a54b876e95aab12a7279e1918b3fb086c560ee6b114c1a0e170bea"
Oct 30 19:36:18.847934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.847898    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"]
Oct 30 19:36:18.854754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.854728    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.883750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.883719    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"]
Oct 30 19:36:18.884540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.884511    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.884720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.884696    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.884833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.884817    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.888203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.888180    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:18.888308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.888241    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:36:18.991587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.991551    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.992323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.992291    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.992477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.992386    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.993339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.993319    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:18.995492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:18.995467    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:19.011520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:19.011495    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\") pod \"maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:19.184942 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:19.184920    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:19.491736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:19.491711    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"]
Oct 30 19:36:19.492868 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:36:19.492841    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fd27ca_8287_4840_989e_2d08a72f9f92.slice/crio-fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f WatchSource:0}: Error finding container fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f: Status 404 returned error can't find the container with id fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f
Oct 30 19:36:19.599515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:19.599476    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv" event={"ID":"e8fd27ca-8287-4840-989e-2d08a72f9f92","Type":"ContainerStarted","Data":"fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f"}
Oct 30 19:36:20.650671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:20.647144    4320 generic.go:356] "Generic (PLEG): container finished" podID="e8fd27ca-8287-4840-989e-2d08a72f9f92" containerID="7a923cc1f914b6bb36306d0f52eeac42f601ef79333ebf88af925238beedc505" exitCode=0
Oct 30 19:36:20.650671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:20.647383    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv" event={"ID":"e8fd27ca-8287-4840-989e-2d08a72f9f92","Type":"ContainerDied","Data":"7a923cc1f914b6bb36306d0f52eeac42f601ef79333ebf88af925238beedc505"}
Oct 30 19:36:21.841789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.841765    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:21.948919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.948826    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\") pod \"e8fd27ca-8287-4840-989e-2d08a72f9f92\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") "
Oct 30 19:36:21.949296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.949269    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca" pod "e8fd27ca-8287-4840-989e-2d08a72f9f92" (UID: "e8fd27ca-8287-4840-989e-2d08a72f9f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:21.950711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.949432    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\") pod \"e8fd27ca-8287-4840-989e-2d08a72f9f92\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") "
Oct 30 19:36:21.950711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.949761    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\") pod \"e8fd27ca-8287-4840-989e-2d08a72f9f92\" (UID: \"e8fd27ca-8287-4840-989e-2d08a72f9f92\") "
Oct 30 19:36:21.957442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.957012    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:36:21.967982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.966653    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w" pod "e8fd27ca-8287-4840-989e-2d08a72f9f92" (UID: "e8fd27ca-8287-4840-989e-2d08a72f9f92"). InnerVolumeSpecName "kube-api-access-bhk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:21.970124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:21.970060    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca" pod "e8fd27ca-8287-4840-989e-2d08a72f9f92" (UID: "e8fd27ca-8287-4840-989e-2d08a72f9f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:22.058805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:22.058771    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/e8fd27ca-8287-4840-989e-2d08a72f9f92-kube-api-access-bhk2w\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.058805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:22.058801    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fd27ca-8287-4840-989e-2d08a72f9f92-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:36:22.674834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:22.674798    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv" event={"ID":"e8fd27ca-8287-4840-989e-2d08a72f9f92","Type":"ContainerDied","Data":"fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f"}
Oct 30 19:36:22.675005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:22.674841    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"
Oct 30 19:36:22.675005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:22.674870    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9f85163ceaceed343205d1e1aee8416d3770d6f1ad79e02418c04ddd9ce39f"
Oct 30 19:36:24.867253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:24.867003    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:36:25.924427 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.924395    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 19:36:25.925308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.925284    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e8fd27ca-8287-4840-989e-2d08a72f9f92" containerName="revoke-keys"
Oct 30 19:36:25.925492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.925473    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fd27ca-8287-4840-989e-2d08a72f9f92" containerName="revoke-keys"
Oct 30 19:36:25.925909 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.925885    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fd27ca-8287-4840-989e-2d08a72f9f92" containerName="revoke-keys"
Oct 30 19:36:25.930726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.930702    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:25.960536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:25.960508    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 19:36:26.020095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020055    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020152    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020198    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020218    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8pr\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020256    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020407    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020477    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020545    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.020646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.020590    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122472    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122529    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122570    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122639    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122747    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122803    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122833    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.122886 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122856    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8pr\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122908    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122922    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.122943    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.123282    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.123374    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.123719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.123691    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.124950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.124929    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.125362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.125343    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.143013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.142988    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.143135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.143112    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8pr\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.263676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.258521    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:26.936125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.927086    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 19:36:26.952797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.952282    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:26.952797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.952374    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:26.952797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:26.952423    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:27.715937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:27.715902    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" event={"ID":"8f113b99-eb42-4364-b901-37eead38c18f","Type":"ContainerStarted","Data":"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"}
Oct 30 19:36:27.716116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:27.715960    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" event={"ID":"8f113b99-eb42-4364-b901-37eead38c18f","Type":"ContainerStarted","Data":"385c4cfa6112578911dd00bb1f361c1c17969ad4266086172c65241c7cffe620"}
Oct 30 19:36:28.258989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:28.258938    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:28.260304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:28.260279    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:36:28.260414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:28.260347    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 19:36:28.649789 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:36:28.649750    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:36:29.259553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:29.259522    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:36:29.259962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:29.259572    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 19:36:30.262320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:30.262280    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:36:30.262920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:30.262886    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 19:36:31.259857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:31.259825    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.260028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:31.259871    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 19:36:32.260830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:32.260790    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.261232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:32.260854    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 19:36:33.263852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:33.263824    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:33.264285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:33.264216    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:33.264703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:33.264684    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 19:36:33.332548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:33.332503    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podStartSLOduration=8.33248919 podStartE2EDuration="8.33248919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:27.787759473 +0000 UTC m=+1935.823271964" watchObservedRunningTime="2026-10-30 19:36:33.33248919 +0000 UTC m=+1941.368001654"
Oct 30 19:36:45.715015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:45.714979    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"]
Oct 30 19:36:45.726553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:45.726526    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-891d6b-ncsw9"]
Oct 30 19:36:46.104017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:46.103927    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2492d9f9-60e9-4ffc-b6ab-c53b88d3f672" path="/var/lib/kubelet/pods/2492d9f9-60e9-4ffc-b6ab-c53b88d3f672/volumes"
Oct 30 19:36:58.029823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:58.029789    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"]
Oct 30 19:36:58.040852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:58.040821    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89d641d-zg59z"]
Oct 30 19:36:58.101359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:36:58.101327    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381518d5-5806-4e61-9c0c-d13916d91b51" path="/var/lib/kubelet/pods/381518d5-5806-4e61-9c0c-d13916d91b51/volumes"
Oct 30 19:36:58.645801 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:36:58.645754    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:37:05.166673 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:05.163654    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:37:13.631469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.631434    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"]
Oct 30 19:37:13.636148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.636129    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.654511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.654486    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"]
Oct 30 19:37:13.661837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.661815    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:13.662051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.662029    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:37:13.746909 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.746873    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.747041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.746960    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.747041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.747023    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sdj\" (UniqueName: \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.848899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.848866    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.849041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.848929    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.849041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.848960    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sdj\" (UniqueName: \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.849498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.849477    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.851246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.851224    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.868214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.868194    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sdj\" (UniqueName: \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\") pod \"maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:13.950899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:13.950865    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:14.256693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:14.256667    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"]
Oct 30 19:37:14.257563 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:37:14.257534    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fb57b1_5766_4429_a86d_69e122e7c989.slice/crio-023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65 WatchSource:0}: Error finding container 023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65: Status 404 returned error can't find the container with id 023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65
Oct 30 19:37:15.144038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:15.144007    4320 generic.go:356] "Generic (PLEG): container finished" podID="56fb57b1-5766-4429-a86d-69e122e7c989" containerID="456c01ed981d7a10a237fa266a6119fd90cb6ca399dc5adf1878ee9c2d33bd8f" exitCode=0
Oct 30 19:37:15.144369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:15.144062    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl" event={"ID":"56fb57b1-5766-4429-a86d-69e122e7c989","Type":"ContainerDied","Data":"456c01ed981d7a10a237fa266a6119fd90cb6ca399dc5adf1878ee9c2d33bd8f"}
Oct 30 19:37:15.144369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:15.144108    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl" event={"ID":"56fb57b1-5766-4429-a86d-69e122e7c989","Type":"ContainerStarted","Data":"023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65"}
Oct 30 19:37:16.324142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.324117    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:16.385337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.385295    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\") pod \"56fb57b1-5766-4429-a86d-69e122e7c989\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") "
Oct 30 19:37:16.385516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.385500    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\") pod \"56fb57b1-5766-4429-a86d-69e122e7c989\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") "
Oct 30 19:37:16.385637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.385619    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\" (UniqueName: \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\") pod \"56fb57b1-5766-4429-a86d-69e122e7c989\" (UID: \"56fb57b1-5766-4429-a86d-69e122e7c989\") "
Oct 30 19:37:16.385902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.385872    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca" pod "56fb57b1-5766-4429-a86d-69e122e7c989" (UID: "56fb57b1-5766-4429-a86d-69e122e7c989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:16.386557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.386527    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:16.388039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.388013    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj" pod "56fb57b1-5766-4429-a86d-69e122e7c989" (UID: "56fb57b1-5766-4429-a86d-69e122e7c989"). InnerVolumeSpecName "kube-api-access-29sdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:16.388243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.388217    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca" pod "56fb57b1-5766-4429-a86d-69e122e7c989" (UID: "56fb57b1-5766-4429-a86d-69e122e7c989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:16.487712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.487684    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29sdj\" (UniqueName: \"kubernetes.io/projected/56fb57b1-5766-4429-a86d-69e122e7c989-kube-api-access-29sdj\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:16.487712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:16.487709    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56fb57b1-5766-4429-a86d-69e122e7c989-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:17.162292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:17.162267    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"
Oct 30 19:37:17.162292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:17.162279    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl" event={"ID":"56fb57b1-5766-4429-a86d-69e122e7c989","Type":"ContainerDied","Data":"023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65"}
Oct 30 19:37:17.162491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:17.162319    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="023b4c914565b4cbe24d88dd4957d745425500b01d4264d334f5444f335eef65"
Oct 30 19:37:28.677790 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:37:28.677722    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:37:29.289450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:29.289420    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"]
Oct 30 19:37:29.295321 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:29.295292    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092bc0-fvfdv"]
Oct 30 19:37:30.261360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:30.261322    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fd27ca-8287-4840-989e-2d08a72f9f92" path="/var/lib/kubelet/pods/e8fd27ca-8287-4840-989e-2d08a72f9f92/volumes"
Oct 30 19:37:36.881305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.881270    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"]
Oct 30 19:37:36.882096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.882068    4320 cpu_manager.go:414] "containerMap: removing container" podUID="56fb57b1-5766-4429-a86d-69e122e7c989" containerName="revoke-keys"
Oct 30 19:37:36.882096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.882092    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fb57b1-5766-4429-a86d-69e122e7c989" containerName="revoke-keys"
Oct 30 19:37:36.882283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.882247    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fb57b1-5766-4429-a86d-69e122e7c989" containerName="revoke-keys"
Oct 30 19:37:36.885545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.885525    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:36.905095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.905072    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"]
Oct 30 19:37:36.958501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.958467    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:36.958657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.958580    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:37:36.969532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.969508    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:36.969694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.969675    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llxs\" (UniqueName: \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:36.969812 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:36.969796    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.070739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.070715    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llxs\" (UniqueName: \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.070855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.070796    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.070921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.070909    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.071507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.071482    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.073158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.073135    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.090114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.090088    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llxs\" (UniqueName: \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\") pod \"maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.198912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.198892    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:37.502511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:37.502479    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"]
Oct 30 19:37:37.503240 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:37:37.503214    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d408f8_226c_4d9f_a2d8_cf1fca464d47.slice/crio-ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6 WatchSource:0}: Error finding container ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6: Status 404 returned error can't find the container with id ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6
Oct 30 19:37:38.422758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:38.422725    4320 generic.go:356] "Generic (PLEG): container finished" podID="c6d408f8-226c-4d9f-a2d8-cf1fca464d47" containerID="eae06a505fb6c415d64548930e6422326196eaced14ab5d9bd38af26be83b65c" exitCode=0
Oct 30 19:37:38.423069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:38.422803    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv" event={"ID":"c6d408f8-226c-4d9f-a2d8-cf1fca464d47","Type":"ContainerDied","Data":"eae06a505fb6c415d64548930e6422326196eaced14ab5d9bd38af26be83b65c"}
Oct 30 19:37:38.423069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:38.422853    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv" event={"ID":"c6d408f8-226c-4d9f-a2d8-cf1fca464d47","Type":"ContainerStarted","Data":"ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6"}
Oct 30 19:37:39.034219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.034186    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"]
Oct 30 19:37:39.045736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.045711    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e5c2c3-th6gb"]
Oct 30 19:37:39.617801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.617772    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:39.711643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.711549    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\") pod \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") "
Oct 30 19:37:39.711798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.711671    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\" (UniqueName: \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\") pod \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") "
Oct 30 19:37:39.711798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.711726    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\") pod \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\" (UID: \"c6d408f8-226c-4d9f-a2d8-cf1fca464d47\") "
Oct 30 19:37:39.711979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.711952    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca" pod "c6d408f8-226c-4d9f-a2d8-cf1fca464d47" (UID: "c6d408f8-226c-4d9f-a2d8-cf1fca464d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:39.712536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.712510    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:39.713946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.713917    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs" pod "c6d408f8-226c-4d9f-a2d8-cf1fca464d47" (UID: "c6d408f8-226c-4d9f-a2d8-cf1fca464d47"). InnerVolumeSpecName "kube-api-access-2llxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:39.714066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.714031    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca" pod "c6d408f8-226c-4d9f-a2d8-cf1fca464d47" (UID: "c6d408f8-226c-4d9f-a2d8-cf1fca464d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:39.813902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.813875    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:39.813902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:39.813900    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llxs\" (UniqueName: \"kubernetes.io/projected/c6d408f8-226c-4d9f-a2d8-cf1fca464d47-kube-api-access-2llxs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:40.097736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:40.097669    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c23e57-07fa-4911-9cdd-d3a62f8881b4" path="/var/lib/kubelet/pods/22c23e57-07fa-4911-9cdd-d3a62f8881b4/volumes"
Oct 30 19:37:40.441028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:40.440996    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv" event={"ID":"c6d408f8-226c-4d9f-a2d8-cf1fca464d47","Type":"ContainerDied","Data":"ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6"}
Oct 30 19:37:40.441185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:40.441048    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecfd9c4a2cc258062cfa987feadb8571389a480cd5c15541895d58afe600c2f6"
Oct 30 19:37:40.441185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:40.441094    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"
Oct 30 19:37:48.033667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:48.033612    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"]
Oct 30 19:37:48.061851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:48.061795    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7ce29ce-x4pcx"]
Oct 30 19:37:48.101416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:48.101357    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45faa3ea-97fd-4e91-a462-9dc1169caba5" path="/var/lib/kubelet/pods/45faa3ea-97fd-4e91-a462-9dc1169caba5/volumes"
Oct 30 19:37:48.752341 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:48.752308    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:37:51.558951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.558915    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"]
Oct 30 19:37:51.559789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.559766    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c6d408f8-226c-4d9f-a2d8-cf1fca464d47" containerName="revoke-keys"
Oct 30 19:37:51.559789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.559786    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d408f8-226c-4d9f-a2d8-cf1fca464d47" containerName="revoke-keys"
Oct 30 19:37:51.559974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.559961    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d408f8-226c-4d9f-a2d8-cf1fca464d47" containerName="revoke-keys"
Oct 30 19:37:51.564903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.564886    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.580996 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.580972    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"]
Oct 30 19:37:51.588256 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.588235    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:51.588374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.588348    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:37:51.608560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.608534    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkf2\" (UniqueName: \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.608681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.608647    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.608775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.608758    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.710204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.710171    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.710365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.710232    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.710538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.710517    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkf2\" (UniqueName: \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.711008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.710984    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.712610 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.712581    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.729759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.729730    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkf2\" (UniqueName: \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\") pod \"maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:51.880270 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:51.880207    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:52.188257 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:52.188229    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"]
Oct 30 19:37:52.190548 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:37:52.190520    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84d872_1d8a_4448_ba60_a2cfb10b2609.slice/crio-7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519 WatchSource:0}: Error finding container 7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519: Status 404 returned error can't find the container with id 7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519
Oct 30 19:37:52.551920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:52.551888    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55" event={"ID":"7e84d872-1d8a-4448-ba60-a2cfb10b2609","Type":"ContainerStarted","Data":"9b64defd0628bbea4a6a1dfc3e3831c894485547e7384ce8ac9239a48b9a477a"}
Oct 30 19:37:52.552038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:52.551941    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55" event={"ID":"7e84d872-1d8a-4448-ba60-a2cfb10b2609","Type":"ContainerStarted","Data":"7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519"}
Oct 30 19:37:52.591011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:52.590959    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55" podStartSLOduration=1.5909432749999999 podStartE2EDuration="1.590943275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:52.589360132 +0000 UTC m=+2020.624872596" watchObservedRunningTime="2026-10-30 19:37:52.590943275 +0000 UTC m=+2020.626455742"
Oct 30 19:37:53.560956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:53.560913    4320 generic.go:356] "Generic (PLEG): container finished" podID="7e84d872-1d8a-4448-ba60-a2cfb10b2609" containerID="9b64defd0628bbea4a6a1dfc3e3831c894485547e7384ce8ac9239a48b9a477a" exitCode=0
Oct 30 19:37:53.561143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:53.560962    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55" event={"ID":"7e84d872-1d8a-4448-ba60-a2cfb10b2609","Type":"ContainerDied","Data":"9b64defd0628bbea4a6a1dfc3e3831c894485547e7384ce8ac9239a48b9a477a"}
Oct 30 19:37:54.738883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.738858    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:54.858468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.858361    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\" (UniqueName: \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\") pod \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") "
Oct 30 19:37:54.858468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.858441    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\") pod \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") "
Oct 30 19:37:54.858788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.858573    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\") pod \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\" (UID: \"7e84d872-1d8a-4448-ba60-a2cfb10b2609\") "
Oct 30 19:37:54.858995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.858967    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca" pod "7e84d872-1d8a-4448-ba60-a2cfb10b2609" (UID: "7e84d872-1d8a-4448-ba60-a2cfb10b2609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:54.859308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.859290    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:54.860433 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.860411    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2" pod "7e84d872-1d8a-4448-ba60-a2cfb10b2609" (UID: "7e84d872-1d8a-4448-ba60-a2cfb10b2609"). InnerVolumeSpecName "kube-api-access-lvkf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:54.860513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.860436    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca" pod "7e84d872-1d8a-4448-ba60-a2cfb10b2609" (UID: "7e84d872-1d8a-4448-ba60-a2cfb10b2609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:54.961121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.961089    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkf2\" (UniqueName: \"kubernetes.io/projected/7e84d872-1d8a-4448-ba60-a2cfb10b2609-kube-api-access-lvkf2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:54.961121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:54.961118    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84d872-1d8a-4448-ba60-a2cfb10b2609-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:37:55.582889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:55.582859    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"
Oct 30 19:37:55.582889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:55.582876    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55" event={"ID":"7e84d872-1d8a-4448-ba60-a2cfb10b2609","Type":"ContainerDied","Data":"7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519"}
Oct 30 19:37:55.583104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:37:55.582927    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c48440602c554ee03ef843af5c91bded69a0639a4770ea89c189bc668be3519"
Oct 30 19:37:58.644297 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:37:58.644257    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:38:03.737732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:03.737696    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"]
Oct 30 19:38:03.750645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:03.750615    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ccf8bd61-2kq55"]
Oct 30 19:38:04.099820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:04.099743    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84d872-1d8a-4448-ba60-a2cfb10b2609" path="/var/lib/kubelet/pods/7e84d872-1d8a-4448-ba60-a2cfb10b2609/volumes"
Oct 30 19:38:13.051687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:13.051653    4320 scope.go:122] "RemoveContainer" containerID="ab2bbbd3c5374a7c5946210167473b5eb4a8c2354d2fb0680ba71f5792a79cfb"
Oct 30 19:38:14.374668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:14.374635    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"]
Oct 30 19:38:14.393611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:14.393563    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7d9e82-q26wl"]
Oct 30 19:38:16.097352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:16.097316    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fb57b1-5766-4429-a86d-69e122e7c989" path="/var/lib/kubelet/pods/56fb57b1-5766-4429-a86d-69e122e7c989/volumes"
Oct 30 19:38:28.646527 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:38:28.646489    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:38:33.895514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:33.895480    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:38:40.617945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:40.617910    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:38:40.618502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:40.618111    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy" containerID="cri-o://82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf" gracePeriod=30
Oct 30 19:38:46.021897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.021859    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:38:46.097291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.097251    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.223134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.223037    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.223705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.223678    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.224008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.223923    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.230711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.230058    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.231114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.231089    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.231535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.231507    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.233655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.233574    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.234498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.234478    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\") pod \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\" (UID: \"80e05e26-57fc-45cf-8ada-b1d41a642ed2\") "
Oct 30 19:38:46.235762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.098914    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:46.238493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.229799    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:46.238677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.234079    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:46.238795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.237707    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:46.238980 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.238447    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.287632    4320 generic.go:356] "Generic (PLEG): container finished" podID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerID="82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf" exitCode=0
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.287895    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" event={"ID":"80e05e26-57fc-45cf-8ada-b1d41a642ed2","Type":"ContainerDied","Data":"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"}
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.287956    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88" event={"ID":"80e05e26-57fc-45cf-8ada-b1d41a642ed2","Type":"ContainerDied","Data":"107fbb8ca6c3e42a321009f35288271ffb862810811c393f2235bd2b1f8e48dd"}
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.287987    4320 scope.go:122] "RemoveContainer" containerID="82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.288145    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"
Oct 30 19:38:46.298645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.292330    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:46.328082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.327607    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "kube-api-access-59hv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:46.329658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.329529    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:46.331831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.331790    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo" pod "80e05e26-57fc-45cf-8ada-b1d41a642ed2" (UID: "80e05e26-57fc-45cf-8ada-b1d41a642ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:46.338379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.338325    4320 scope.go:122] "RemoveContainer" containerID="82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"
Oct 30 19:38:46.339003 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:38:46.338970    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf\": container with ID starting with 82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf not found: ID does not exist" containerID="82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"
Oct 30 19:38:46.339130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.339016    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf"} err="failed to get container status \"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf\": rpc error: code = NotFound desc = could not find container \"82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf\": container with ID starting with 82db86da6ecb263d09b7ce778ba6a3f907e95e22f7c976fe334bdec3d82227bf not found: ID does not exist"
Oct 30 19:38:46.390748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390716    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59hv8\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-kube-api-access-59hv8\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390747    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390761    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390773    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390785    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390804    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390816    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390827    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.390963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.390838    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e05e26-57fc-45cf-8ada-b1d41a642ed2-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:46.683060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.682722    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:38:46.709639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:46.705970    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-2qm88"]
Oct 30 19:38:48.099401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.099363    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" path="/var/lib/kubelet/pods/80e05e26-57fc-45cf-8ada-b1d41a642ed2/volumes"
Oct 30 19:38:48.711585 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.711547    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"]
Oct 30 19:38:48.712283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712265    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7e84d872-1d8a-4448-ba60-a2cfb10b2609" containerName="revoke-keys"
Oct 30 19:38:48.712345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712288    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84d872-1d8a-4448-ba60-a2cfb10b2609" containerName="revoke-keys"
Oct 30 19:38:48.712345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712324    4320 cpu_manager.go:414] "containerMap: removing container" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy"
Oct 30 19:38:48.712345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712330    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy"
Oct 30 19:38:48.712455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712427    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84d872-1d8a-4448-ba60-a2cfb10b2609" containerName="revoke-keys"
Oct 30 19:38:48.712455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.712441    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e05e26-57fc-45cf-8ada-b1d41a642ed2" containerName="istio-proxy"
Oct 30 19:38:48.717580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.717559    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.740107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.740085    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"]
Oct 30 19:38:48.747003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.746981    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:48.747132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.747097    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:38:48.835084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.835043    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.835677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.835151    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77p5\" (UniqueName: \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.835677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.835309    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.936850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.936820    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.937029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.936988    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.937081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.937047    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77p5\" (UniqueName: \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.937479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.937458    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.939449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.939425    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:48.961276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:48.961249    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77p5\" (UniqueName: \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\") pod \"maas-api-revoke-keys-e2e-api-a-d4cb13-pg297\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:49.032186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:49.032128    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:49.338201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:49.338174    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"]
Oct 30 19:38:49.338793 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:38:49.338763    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad5c838_bd7f_424b_b2f9_54f5365831b4.slice/crio-b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c WatchSource:0}: Error finding container b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c: Status 404 returned error can't find the container with id b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c
Oct 30 19:38:50.325932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:50.325901    4320 generic.go:356] "Generic (PLEG): container finished" podID="2ad5c838-bd7f-424b-b2f9-54f5365831b4" containerID="9ff69108019b2a30c26f1572878c3dbde591e5ee8f2ff495ce3dc2e51817fefc" exitCode=0
Oct 30 19:38:50.326077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:50.325971    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297" event={"ID":"2ad5c838-bd7f-424b-b2f9-54f5365831b4","Type":"ContainerDied","Data":"9ff69108019b2a30c26f1572878c3dbde591e5ee8f2ff495ce3dc2e51817fefc"}
Oct 30 19:38:50.326077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:50.326021    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297" event={"ID":"2ad5c838-bd7f-424b-b2f9-54f5365831b4","Type":"ContainerStarted","Data":"b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c"}
Oct 30 19:38:51.501065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.501043    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:51.676622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.676564    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\") pod \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") "
Oct 30 19:38:51.676786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.676737    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\" (UniqueName: \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\") pod \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") "
Oct 30 19:38:51.676888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.676876    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\") pod \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\" (UID: \"2ad5c838-bd7f-424b-b2f9-54f5365831b4\") "
Oct 30 19:38:51.677159 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.677133    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca" pod "2ad5c838-bd7f-424b-b2f9-54f5365831b4" (UID: "2ad5c838-bd7f-424b-b2f9-54f5365831b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:51.677469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.677453    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:51.678643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.678617    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca" pod "2ad5c838-bd7f-424b-b2f9-54f5365831b4" (UID: "2ad5c838-bd7f-424b-b2f9-54f5365831b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:51.678831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.678813    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5" pod "2ad5c838-bd7f-424b-b2f9-54f5365831b4" (UID: "2ad5c838-bd7f-424b-b2f9-54f5365831b4"). InnerVolumeSpecName "kube-api-access-q77p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:51.779114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.779087    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77p5\" (UniqueName: \"kubernetes.io/projected/2ad5c838-bd7f-424b-b2f9-54f5365831b4-kube-api-access-q77p5\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:51.779114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:51.779113    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad5c838-bd7f-424b-b2f9-54f5365831b4-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:38:52.342576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:52.342547    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"
Oct 30 19:38:52.342769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:52.342574    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297" event={"ID":"2ad5c838-bd7f-424b-b2f9-54f5365831b4","Type":"ContainerDied","Data":"b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c"}
Oct 30 19:38:52.342769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:52.342653    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5df869444a461562410e0ea02f2863c3e926d8abc7d417e98e3c73dcd12758c"
Oct 30 19:38:55.685257 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:55.685222    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"]
Oct 30 19:38:55.698719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:55.698664    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b625eb-hs9fv"]
Oct 30 19:38:56.098429 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:38:56.098346    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d408f8-226c-4d9f-a2d8-cf1fca464d47" path="/var/lib/kubelet/pods/c6d408f8-226c-4d9f-a2d8-cf1fca464d47/volumes"
Oct 30 19:38:58.645438 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:38:58.645404    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:39:10.234515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.234472    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"]
Oct 30 19:39:10.237051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.235572    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2ad5c838-bd7f-424b-b2f9-54f5365831b4" containerName="revoke-keys"
Oct 30 19:39:10.237051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.235591    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad5c838-bd7f-424b-b2f9-54f5365831b4" containerName="revoke-keys"
Oct 30 19:39:10.237051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.235760    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad5c838-bd7f-424b-b2f9-54f5365831b4" containerName="revoke-keys"
Oct 30 19:39:10.244028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.243992    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.245562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.245531    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"]
Oct 30 19:39:10.266294 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.266229    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.266476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.266344    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52nd\" (UniqueName: \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.266476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.266386    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.279301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.279276    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:10.279301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.279294    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:39:10.368274 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.368244    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.368439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.368332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52nd\" (UniqueName: \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.368439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.368361    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.369027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.368998    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.370838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.370819    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.387734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.387709    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52nd\" (UniqueName: \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\") pod \"maas-api-revoke-sub-bef61f35-jxbb9\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.569624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.569534    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:10.877805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:10.873205    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"]
Oct 30 19:39:11.602149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:11.602069    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9" event={"ID":"9f765a77-9e2b-4170-90c7-15c0316b0ccb","Type":"ContainerStarted","Data":"39a48d44b0da6af702219290d6f440389a56861855365bebd751e3b83882675d"}
Oct 30 19:39:11.602658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:11.602167    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9" event={"ID":"9f765a77-9e2b-4170-90c7-15c0316b0ccb","Type":"ContainerStarted","Data":"6c5c8a8c1968ae4fcd703e698455da4d54713a5ec520f680123b36f2cc62ed91"}
Oct 30 19:39:11.645586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:11.645534    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9" podStartSLOduration=1.645516827 podStartE2EDuration="1.645516827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:11.641898546 +0000 UTC m=+2099.677411009" watchObservedRunningTime="2026-10-30 19:39:11.645516827 +0000 UTC m=+2099.681029295"
Oct 30 19:39:12.611188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:12.611146    4320 generic.go:356] "Generic (PLEG): container finished" podID="9f765a77-9e2b-4170-90c7-15c0316b0ccb" containerID="39a48d44b0da6af702219290d6f440389a56861855365bebd751e3b83882675d" exitCode=0
Oct 30 19:39:12.611743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:12.611204    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9" event={"ID":"9f765a77-9e2b-4170-90c7-15c0316b0ccb","Type":"ContainerDied","Data":"39a48d44b0da6af702219290d6f440389a56861855365bebd751e3b83882675d"}
Oct 30 19:39:13.083304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.083268    4320 scope.go:122] "RemoveContainer" containerID="33eef0ff8498bb728144d90c00767abf1f0f70202be0cb36f0c84faef4ee6642"
Oct 30 19:39:13.096836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.096806    4320 scope.go:122] "RemoveContainer" containerID="563487765ca06ef2e1d946351b4cf2f425ec0c0a6b8eb19f8bc50039f6d8dbcd"
Oct 30 19:39:13.107681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.107663    4320 scope.go:122] "RemoveContainer" containerID="f7f1fa2347783d95983b823f7ac3cda0902fa3fd1e5c7e1add2c9054320d3205"
Oct 30 19:39:13.792390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.792362    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:13.852965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.852938    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\" (UniqueName: \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\") pod \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") "
Oct 30 19:39:13.853117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.853035    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\") pod \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") "
Oct 30 19:39:13.853243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.853228    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\") pod \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\" (UID: \"9f765a77-9e2b-4170-90c7-15c0316b0ccb\") "
Oct 30 19:39:13.853683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.853426    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca" pod "9f765a77-9e2b-4170-90c7-15c0316b0ccb" (UID: "9f765a77-9e2b-4170-90c7-15c0316b0ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:13.854008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.853987    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:13.855808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.855781    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd" pod "9f765a77-9e2b-4170-90c7-15c0316b0ccb" (UID: "9f765a77-9e2b-4170-90c7-15c0316b0ccb"). InnerVolumeSpecName "kube-api-access-m52nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:13.855932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.855875    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca" pod "9f765a77-9e2b-4170-90c7-15c0316b0ccb" (UID: "9f765a77-9e2b-4170-90c7-15c0316b0ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:13.955903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.955879    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m52nd\" (UniqueName: \"kubernetes.io/projected/9f765a77-9e2b-4170-90c7-15c0316b0ccb-kube-api-access-m52nd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:13.955903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:13.955901    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f765a77-9e2b-4170-90c7-15c0316b0ccb-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:14.627545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:14.627514    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"
Oct 30 19:39:14.627732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:14.627544    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9" event={"ID":"9f765a77-9e2b-4170-90c7-15c0316b0ccb","Type":"ContainerDied","Data":"6c5c8a8c1968ae4fcd703e698455da4d54713a5ec520f680123b36f2cc62ed91"}
Oct 30 19:39:14.627732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:14.627634    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c5c8a8c1968ae4fcd703e698455da4d54713a5ec520f680123b36f2cc62ed91"
Oct 30 19:39:23.960401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:23.932534    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:39:28.059624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.058957    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:39:28.059624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.059031    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:28.112735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.082741    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:39:28.112735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.083072    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy" containerID="cri-o://7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030" gracePeriod=30
Oct 30 19:39:28.394684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.377645    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:39:28.395291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.395255    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy" containerID="cri-o://a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299" gracePeriod=30
Oct 30 19:39:28.395619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.394957    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:28.485717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:28.485673    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:28.756679 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:28.742780    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:39:31.032698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:31.032660    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"]
Oct 30 19:39:31.046354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:31.046323    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc4e2b-mxzl7"]
Oct 30 19:39:32.099030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:32.098999    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36c19bb-23c7-4740-9418-a894bd695753" path="/var/lib/kubelet/pods/f36c19bb-23c7-4740-9418-a894bd695753/volumes"
Oct 30 19:39:33.000539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.000477    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:39:33.000748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.000679    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy" containerID="cri-o://8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6" gracePeriod=30
Oct 30 19:39:33.013908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.013880    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:39:33.014095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.014048    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy" containerID="cri-o://121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8" gracePeriod=30
Oct 30 19:39:33.574665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.574637    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:39:33.607969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.607942    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:39:33.690426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690390    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.690625 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690450    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.690625 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690537    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.690755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690639    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.690755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690712    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.690867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690808    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.690867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690821    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.690978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690867    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.690978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690890    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.690978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690933    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.690978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.690973    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691049    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691100    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691111    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691112    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691138    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:33.691186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691172    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691244    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691280    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691280    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691353    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691355    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691428    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\") pod \"9580643d-b813-497c-b82c-a428837ec65e\" (UID: \"9580643d-b813-497c-b82c-a428837ec65e\") "
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691435    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:33.691502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691491    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\") pod \"59f2b7b8-ec87-4055-8257-7074925e1b63\" (UID: \"59f2b7b8-ec87-4055-8257-7074925e1b63\") "
Oct 30 19:39:33.691939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691512    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691667    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.691939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.691793    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.692329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692302    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692337    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692353    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f2b7b8-ec87-4055-8257-7074925e1b63-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692367    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692379    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692392    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9580643d-b813-497c-b82c-a428837ec65e-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692405    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692418    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692432    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.692930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.692447    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.694952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.694925    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.696977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.696952    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "kube-api-access-s85sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:33.696977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.696966    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:33.696977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.696982    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "kube-api-access-9w79w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:33.697165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.696995    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token" pod "9580643d-b813-497c-b82c-a428837ec65e" (UID: "9580643d-b813-497c-b82c-a428837ec65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:33.697165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.697007    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:33.697165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.697014    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:33.700842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.700818    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy" pod "59f2b7b8-ec87-4055-8257-7074925e1b63" (UID: "59f2b7b8-ec87-4055-8257-7074925e1b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.793703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793675    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w79w\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-kube-api-access-9w79w\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793702    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85sr\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-kube-api-access-s85sr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793716    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9580643d-b813-497c-b82c-a428837ec65e-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793732    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9580643d-b813-497c-b82c-a428837ec65e-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793746    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793759    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793771    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f2b7b8-ec87-4055-8257-7074925e1b63-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:33.793883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:33.793784    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9580643d-b813-497c-b82c-a428837ec65e-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:34.027318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.027284    4320 generic.go:356] "Generic (PLEG): container finished" podID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerID="7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030" exitCode=0
Oct 30 19:39:34.027516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.027327    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"
Oct 30 19:39:34.027516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.027374    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" event={"ID":"59f2b7b8-ec87-4055-8257-7074925e1b63","Type":"ContainerDied","Data":"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"}
Oct 30 19:39:34.027516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.027442    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk" event={"ID":"59f2b7b8-ec87-4055-8257-7074925e1b63","Type":"ContainerDied","Data":"1c7ebfcdeee552af0899756f21599f6e17b1fee69fd43481de07a1ff0cbaa9e0"}
Oct 30 19:39:34.027516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.027475    4320 scope.go:122] "RemoveContainer" containerID="7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"
Oct 30 19:39:34.029033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.029009    4320 generic.go:356] "Generic (PLEG): container finished" podID="9580643d-b813-497c-b82c-a428837ec65e" containerID="a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299" exitCode=0
Oct 30 19:39:34.029105 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.029058    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"
Oct 30 19:39:34.029105 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.029083    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" event={"ID":"9580643d-b813-497c-b82c-a428837ec65e","Type":"ContainerDied","Data":"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"}
Oct 30 19:39:34.029194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.029130    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8" event={"ID":"9580643d-b813-497c-b82c-a428837ec65e","Type":"ContainerDied","Data":"f575e163bc91432cdad3cd5a555cee2aa6815308115b9b7b1bc1d0d52af70098"}
Oct 30 19:39:34.052069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.052048    4320 scope.go:122] "RemoveContainer" containerID="7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"
Oct 30 19:39:34.052374 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:34.052358    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030\": container with ID starting with 7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030 not found: ID does not exist" containerID="7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"
Oct 30 19:39:34.052422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.052383    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030"} err="failed to get container status \"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030\": rpc error: code = NotFound desc = could not find container \"7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030\": container with ID starting with 7f6a8adab6bdfee945a7daa6a7417e72a9739f27d976bc66265780c5c2237030 not found: ID does not exist"
Oct 30 19:39:34.052422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.052400    4320 scope.go:122] "RemoveContainer" containerID="a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"
Oct 30 19:39:34.062193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.062172    4320 scope.go:122] "RemoveContainer" containerID="a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"
Oct 30 19:39:34.062466 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:34.062449    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299\": container with ID starting with a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299 not found: ID does not exist" containerID="a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"
Oct 30 19:39:34.062517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.062473    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299"} err="failed to get container status \"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299\": rpc error: code = NotFound desc = could not find container \"a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299\": container with ID starting with a8da090f86fadd5d1528c701db9e912f0670b6291e0fbcbb76f2b870106af299 not found: ID does not exist"
Oct 30 19:39:34.099519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.099495    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:39:34.105412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.105390    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-kkvtk"]
Oct 30 19:39:34.138648 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.138588    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:39:34.142204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:34.142181    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-2ptv8"]
Oct 30 19:39:36.099570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:36.099535    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" path="/var/lib/kubelet/pods/59f2b7b8-ec87-4055-8257-7074925e1b63/volumes"
Oct 30 19:39:36.100070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:36.100053    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9580643d-b813-497c-b82c-a428837ec65e" path="/var/lib/kubelet/pods/9580643d-b813-497c-b82c-a428837ec65e/volumes"
Oct 30 19:39:38.290089 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.290066    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:39:38.353122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.353096    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:39:38.452478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452445    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.452700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452499    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.452700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452555    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.452700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452626    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.452700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452683    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.453005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452718    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452812    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452892    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.452992    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.453220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453012    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.453220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453059    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453084    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:38.453220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453117    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453218    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.453437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453305    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.453437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453400    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.453584 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453487    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.454335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453782    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.454335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453834    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.454335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.453991    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.454526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.454332    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.454526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.454407    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.454526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.454501    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\") pod \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\" (UID: \"7a61ddab-2049-47ca-8fe6-4ffaabb1251a\") "
Oct 30 19:39:38.454727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.454590    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.454727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.454658    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\") pod \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\" (UID: \"7c71e168-4657-48f9-bc1b-d2b6446ddfaf\") "
Oct 30 19:39:38.455695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.455268    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.455888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.455864    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:38.456428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456402    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456430    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456452    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456467    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456482    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456496    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456538    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.456553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456553    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.457077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456569    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.457077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.456837    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.460384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.460358    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "kube-api-access-mzllh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:38.460494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.460471    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:38.462717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.462551    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:38.463087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.462728    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:38.463087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.462785    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "kube-api-access-2rc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:38.464156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.464020    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy" pod "7a61ddab-2049-47ca-8fe6-4ffaabb1251a" (UID: "7a61ddab-2049-47ca-8fe6-4ffaabb1251a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.464280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.464220    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:38.464843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.464818    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy" pod "7c71e168-4657-48f9-bc1b-d2b6446ddfaf" (UID: "7c71e168-4657-48f9-bc1b-d2b6446ddfaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558195    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558221    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558230    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558241    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzllh\" (UniqueName: \"kubernetes.io/projected/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-kube-api-access-mzllh\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558256    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558269    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558282    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558295    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71e168-4657-48f9-bc1b-d2b6446ddfaf-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:38.558660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:38.558308    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc6s\" (UniqueName: \"kubernetes.io/projected/7a61ddab-2049-47ca-8fe6-4ffaabb1251a-kube-api-access-2rc6s\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:39.068920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.068883    4320 generic.go:356] "Generic (PLEG): container finished" podID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerID="121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8" exitCode=0
Oct 30 19:39:39.069112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.068944    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" event={"ID":"7c71e168-4657-48f9-bc1b-d2b6446ddfaf","Type":"ContainerDied","Data":"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"}
Oct 30 19:39:39.069112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.068968    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"
Oct 30 19:39:39.069112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.069042    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp" event={"ID":"7c71e168-4657-48f9-bc1b-d2b6446ddfaf","Type":"ContainerDied","Data":"6ab507648ce12be648009ee934ef0a972fa7fc27d1b0d574ba300843033cbd77"}
Oct 30 19:39:39.069112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.069083    4320 scope.go:122] "RemoveContainer" containerID="121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"
Oct 30 19:39:39.072271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.072233    4320 generic.go:356] "Generic (PLEG): container finished" podID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerID="8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6" exitCode=0
Oct 30 19:39:39.072413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.072292    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"
Oct 30 19:39:39.072413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.072335    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" event={"ID":"7a61ddab-2049-47ca-8fe6-4ffaabb1251a","Type":"ContainerDied","Data":"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"}
Oct 30 19:39:39.072413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.072367    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj" event={"ID":"7a61ddab-2049-47ca-8fe6-4ffaabb1251a","Type":"ContainerDied","Data":"6b93c64286b8fdbab6faab0a33b9408a414d26dec354f970f6ec91e36bd162e5"}
Oct 30 19:39:39.081297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.081281    4320 scope.go:122] "RemoveContainer" containerID="121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"
Oct 30 19:39:39.081628 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:39.081478    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8\": container with ID starting with 121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8 not found: ID does not exist" containerID="121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"
Oct 30 19:39:39.081628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.081504    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8"} err="failed to get container status \"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8\": rpc error: code = NotFound desc = could not find container \"121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8\": container with ID starting with 121e6ea702ba1e9e433fd5121fdae0f50429863a158c530ec3551214d5e117b8 not found: ID does not exist"
Oct 30 19:39:39.081628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.081530    4320 scope.go:122] "RemoveContainer" containerID="8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"
Oct 30 19:39:39.095432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.095411    4320 scope.go:122] "RemoveContainer" containerID="8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"
Oct 30 19:39:39.095897 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:39.095874    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6\": container with ID starting with 8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6 not found: ID does not exist" containerID="8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"
Oct 30 19:39:39.095974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.095903    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6"} err="failed to get container status \"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6\": rpc error: code = NotFound desc = could not find container \"8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6\": container with ID starting with 8849ed664e820648f69368f38383859ff47e2eeef2c5ba38204b6acfd2be22c6 not found: ID does not exist"
Oct 30 19:39:39.123307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.123270    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:39:39.126354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.126333    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-ckxwp"]
Oct 30 19:39:39.161944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.161923    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:39:39.166268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:39.166249    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-q5svj"]
Oct 30 19:39:40.096727 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:40.096699    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" path="/var/lib/kubelet/pods/7a61ddab-2049-47ca-8fe6-4ffaabb1251a/volumes"
Oct 30 19:39:40.097195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:40.097180    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" path="/var/lib/kubelet/pods/7c71e168-4657-48f9-bc1b-d2b6446ddfaf/volumes"
Oct 30 19:39:40.803676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:40.803643    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:39:40.803904 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:40.803880    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy" containerID="cri-o://df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963" gracePeriod=30
Oct 30 19:39:46.080180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.080123    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:39:46.135891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.135835    4320 generic.go:356] "Generic (PLEG): container finished" podID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerID="df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963" exitCode=0
Oct 30 19:39:46.136059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.135895    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"
Oct 30 19:39:46.136059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.136003    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" event={"ID":"6882265c-10fe-49e8-ba44-9696912dd5a0","Type":"ContainerDied","Data":"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"}
Oct 30 19:39:46.136059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.136047    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7" event={"ID":"6882265c-10fe-49e8-ba44-9696912dd5a0","Type":"ContainerDied","Data":"774f958aad9a52a51cd153913cafd7fa615930b66abc6e110caba8f90e2be5ce"}
Oct 30 19:39:46.136277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.136076    4320 scope.go:122] "RemoveContainer" containerID="df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"
Oct 30 19:39:46.165793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.165755    4320 scope.go:122] "RemoveContainer" containerID="df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166129    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:46.166170    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963\": container with ID starting with df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963 not found: ID does not exist" containerID="df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166190    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166207    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963"} err="failed to get container status \"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963\": rpc error: code = NotFound desc = could not find container \"df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963\": container with ID starting with df8488ab22c120b3581d634bd716d4e285ad982075f7ac88a4556d26222f9963 not found: ID does not exist"
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166371    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166465    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166544    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.166994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.166579    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.167587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167163    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.167587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167187    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:46.167587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167248    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.167587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167257    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:46.167587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167307    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\") pod \"6882265c-10fe-49e8-ba44-9696912dd5a0\" (UID: \"6882265c-10fe-49e8-ba44-9696912dd5a0\") "
Oct 30 19:39:46.167877 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167836    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:46.167940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.167889    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:46.168154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.168132    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:46.169092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.168998    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6882265c-10fe-49e8-ba44-9696912dd5a0-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.169092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.169028    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.169092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.169042    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.169092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.169055    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.169092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.169070    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.180798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.180761    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:46.180963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.180941    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "kube-api-access-qcp8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:46.181087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.181068    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:46.182806 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.182785    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo" pod "6882265c-10fe-49e8-ba44-9696912dd5a0" (UID: "6882265c-10fe-49e8-ba44-9696912dd5a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:46.270793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.270761    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.270793 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.270793    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.270989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.270803    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcp8q\" (UniqueName: \"kubernetes.io/projected/6882265c-10fe-49e8-ba44-9696912dd5a0-kube-api-access-qcp8q\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.270989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.270812    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6882265c-10fe-49e8-ba44-9696912dd5a0-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:39:46.483666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.483633    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:39:46.492001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:46.489628    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-jtqc7"]
Oct 30 19:39:47.421150 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421110    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:39:47.421842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421823    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy"
Oct 30 19:39:47.421842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421842    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421870    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421875    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421890    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421895    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421904    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421908    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421921    4320 cpu_manager.go:414] "containerMap: removing container" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421926    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421934    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9f765a77-9e2b-4170-90c7-15c0316b0ccb" containerName="revoke-keys"
Oct 30 19:39:47.421971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.421938    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f765a77-9e2b-4170-90c7-15c0316b0ccb" containerName="revoke-keys"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422025    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9580643d-b813-497c-b82c-a428837ec65e" containerName="istio-proxy"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422034    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71e168-4657-48f9-bc1b-d2b6446ddfaf" containerName="istio-proxy"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422042    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a61ddab-2049-47ca-8fe6-4ffaabb1251a" containerName="istio-proxy"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422049    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f2b7b8-ec87-4055-8257-7074925e1b63" containerName="istio-proxy"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422055    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f765a77-9e2b-4170-90c7-15c0316b0ccb" containerName="revoke-keys"
Oct 30 19:39:47.422418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.422063    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" containerName="istio-proxy"
Oct 30 19:39:47.427222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.427201    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.463878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.463583    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-614c02-serving-cert\""
Oct 30 19:39:47.471736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.471706    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:39:47.492218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.492186    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.492372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.492244    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf268\" (UniqueName: \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.492495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.492470    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.492644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.492586    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.594634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.594578    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.594781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.594656    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf268\" (UniqueName: \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.594781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.594774    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.594915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.594821    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.595250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.595224    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.597260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.597237    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.597378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.597294    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.612787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.612768    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf268\" (UniqueName: \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\") pod \"maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:47.742109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:47.741743    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:48.054632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:48.054587    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:39:48.055669 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:39:48.055642    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ba2e88_aef6_4a17_8489_0b409eceab70.slice/crio-bfd6d293a0c34919c121ccd6a799b7a49e6cabba2c8c692b2e03635cdef92cd6 WatchSource:0}: Error finding container bfd6d293a0c34919c121ccd6a799b7a49e6cabba2c8c692b2e03635cdef92cd6: Status 404 returned error can't find the container with id bfd6d293a0c34919c121ccd6a799b7a49e6cabba2c8c692b2e03635cdef92cd6
Oct 30 19:39:48.057070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:48.057050    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:39:48.099012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:48.098987    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6882265c-10fe-49e8-ba44-9696912dd5a0" path="/var/lib/kubelet/pods/6882265c-10fe-49e8-ba44-9696912dd5a0/volumes"
Oct 30 19:39:48.154405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:48.154375    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" event={"ID":"99ba2e88-aef6-4a17-8489-0b409eceab70","Type":"ContainerStarted","Data":"bfd6d293a0c34919c121ccd6a799b7a49e6cabba2c8c692b2e03635cdef92cd6"}
Oct 30 19:39:49.162792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:49.162761    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" event={"ID":"99ba2e88-aef6-4a17-8489-0b409eceab70","Type":"ContainerStarted","Data":"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"}
Oct 30 19:39:49.163179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:49.162846    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:49.209721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:49.209663    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" podStartSLOduration=1.881050084 podStartE2EDuration="2.209650226s" totalImagesPullingTime="328.600142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:49.205995948 +0000 UTC m=+2137.241508412" watchObservedRunningTime="2026-10-30 19:39:49.209650226 +0000 UTC m=+2137.245162691"
Oct 30 19:39:51.959590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:51.959554    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:39:51.964204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:51.964177    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.000558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.000524    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:39:52.009290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.009254    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-469724-openshift-default-dockercfg-5dzqn\""
Oct 30 19:39:52.058992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.058964    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059017    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059085    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059101    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059193    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059334    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059373    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059400    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6r2\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.059502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.059476    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.160620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.160527    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.160827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.160632    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.160827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.160663    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.160827 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.160710    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.161013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.160930    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.161072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.161009    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.161072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.161060    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6r2\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.161178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.161097    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.161178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.161153    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.193011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.161686    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.193011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.175644    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.193011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.175933    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.193011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.176172    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.193011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.176496    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.196654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.194233    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.247662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.244332    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6r2\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.296627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.290235    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.296627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.293911    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\") pod \"e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") " pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.581658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.581572    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:52.907317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.907286    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:39:52.909290 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:39:52.909263    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7465b832_8cb3_4d74_9604_2e66a10a5738.slice/crio-bee26b8bcd458dc56ab2a8e5dd1a0f328066a75077ceda637e3fd62447fc5aed WatchSource:0}: Error finding container bee26b8bcd458dc56ab2a8e5dd1a0f328066a75077ceda637e3fd62447fc5aed: Status 404 returned error can't find the container with id bee26b8bcd458dc56ab2a8e5dd1a0f328066a75077ceda637e3fd62447fc5aed
Oct 30 19:39:52.913101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.913064    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:52.913208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.913146    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:52.913208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:52.913196    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:53.233502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:53.233409    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerStarted","Data":"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"}
Oct 30 19:39:53.233502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:53.233474    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerStarted","Data":"bee26b8bcd458dc56ab2a8e5dd1a0f328066a75077ceda637e3fd62447fc5aed"}
Oct 30 19:39:53.275574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:53.275514    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podStartSLOduration=2.275496863 podStartE2EDuration="2.275496863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:53.273983475 +0000 UTC m=+2141.309495940" watchObservedRunningTime="2026-10-30 19:39:53.275496863 +0000 UTC m=+2141.311009327"
Oct 30 19:39:53.588899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:53.588809    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:53.598458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:53.598426    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:54.241138 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:54.241103    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:54.242018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:54.241999    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:39:55.175008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:55.174978    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:39:57.199697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.199661    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"]
Oct 30 19:39:57.204275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.204255    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.221368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.221344    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"]
Oct 30 19:39:57.228653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.228632    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:57.228803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.228672    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:39:57.356205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.356175    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgb2d\" (UniqueName: \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.356394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.356378    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.356440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.356412    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.457263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.457189    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.457263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.457229    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.457479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.457341    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgb2d\" (UniqueName: \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.457931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.457906    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.459519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.459496    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.476245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.476221    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgb2d\" (UniqueName: \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\") pod \"maas-api-revoke-sub-608d9fce-4fbmg\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.520477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.520452    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:39:57.825453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:57.825421    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"]
Oct 30 19:39:57.826390 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:39:57.826364    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4c4c1d_7304_47ed_8394_67ed6908942d.slice/crio-edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50 WatchSource:0}: Error finding container edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50: Status 404 returned error can't find the container with id edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50
Oct 30 19:39:58.280068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:58.280040    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg" event={"ID":"dc4c4c1d-7304-47ed-8394-67ed6908942d","Type":"ContainerStarted","Data":"edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50"}
Oct 30 19:39:58.658725 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:39:58.658680    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:39:59.288496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:59.288403    4320 generic.go:356] "Generic (PLEG): container finished" podID="dc4c4c1d-7304-47ed-8394-67ed6908942d" containerID="2cfd2db527f9919ac5779ca2d57b2246d8a350a357d9f7cc39186010822a5d49" exitCode=0
Oct 30 19:39:59.288890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:39:59.288497    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg" event={"ID":"dc4c4c1d-7304-47ed-8394-67ed6908942d","Type":"ContainerDied","Data":"2cfd2db527f9919ac5779ca2d57b2246d8a350a357d9f7cc39186010822a5d49"}
Oct 30 19:40:00.472024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.471996    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:40:00.599162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.599092    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\") pod \"dc4c4c1d-7304-47ed-8394-67ed6908942d\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") "
Oct 30 19:40:00.599304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.599174    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\" (UniqueName: \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\") pod \"dc4c4c1d-7304-47ed-8394-67ed6908942d\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") "
Oct 30 19:40:00.599304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.599279    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\") pod \"dc4c4c1d-7304-47ed-8394-67ed6908942d\" (UID: \"dc4c4c1d-7304-47ed-8394-67ed6908942d\") "
Oct 30 19:40:00.599470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.599443    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca" pod "dc4c4c1d-7304-47ed-8394-67ed6908942d" (UID: "dc4c4c1d-7304-47ed-8394-67ed6908942d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:00.599878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.599858    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:00.601223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.601196    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d" pod "dc4c4c1d-7304-47ed-8394-67ed6908942d" (UID: "dc4c4c1d-7304-47ed-8394-67ed6908942d"). InnerVolumeSpecName "kube-api-access-hgb2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:00.601424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.601404    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca" pod "dc4c4c1d-7304-47ed-8394-67ed6908942d" (UID: "dc4c4c1d-7304-47ed-8394-67ed6908942d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:00.701359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.701330    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgb2d\" (UniqueName: \"kubernetes.io/projected/dc4c4c1d-7304-47ed-8394-67ed6908942d-kube-api-access-hgb2d\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:00.701359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:00.701358    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c4c1d-7304-47ed-8394-67ed6908942d-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:01.314168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:01.314132    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"
Oct 30 19:40:01.314343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:01.314168    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg" event={"ID":"dc4c4c1d-7304-47ed-8394-67ed6908942d","Type":"ContainerDied","Data":"edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50"}
Oct 30 19:40:01.314343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:01.314214    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb3bda81427291e129931a39a916a7b6e4813d111c357c75807daa17477ce50"
Oct 30 19:40:05.551127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.551084    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:40:05.552144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.552119    4320 cpu_manager.go:414] "containerMap: removing container" podUID="dc4c4c1d-7304-47ed-8394-67ed6908942d" containerName="revoke-keys"
Oct 30 19:40:05.552144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.552144    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4c4c1d-7304-47ed-8394-67ed6908942d" containerName="revoke-keys"
Oct 30 19:40:05.552377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.552364    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4c4c1d-7304-47ed-8394-67ed6908942d" containerName="revoke-keys"
Oct 30 19:40:05.557250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.557229    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.576712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576678    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.576826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576730    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.576826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.576826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576819    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnh7\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.577014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576873    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.577014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.576973    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.577014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.577009    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.577180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.577045    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.577180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.577100    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.586592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.586560    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-09feeb-openshift-default-dockercfg-8xm7w\""
Oct 30 19:40:05.586804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.586774    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:40:05.678467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678433    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678572    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678611    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678638    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678666    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnh7\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678726    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678854    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678894    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.678963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.678931    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.679275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.679221    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.679408 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.679383    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.679477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.679385    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.679659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.679632    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.680125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.680099    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.681360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.681336    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.681478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.681399    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.697920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.697873    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.698518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.698499    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnh7\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\") pod \"e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") " pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:05.909663 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:05.908057    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:06.177615 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.177517    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:40:06.189738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.189694    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:40:06.192438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.191772    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.206724 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.206514    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a7b7bc-serving-cert\""
Oct 30 19:40:06.232816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.225691    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.232816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.226000    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.232816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.226137    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.232816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.226184    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.328037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.328005    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.328231 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:40:06.328169    4320 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-serving-cert: secret "maas-api-e2e-shared-a-w3-a7b7bc-serving-cert" not found
Oct 30 19:40:06.328231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.328200    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.328361 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:40:06.328254    4320 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls podName:55f66ef3-600e-4a16-b61d-f255bb0b943c nodeName:}" failed. No retries permitted until 2026-10-30 19:40:06.828234183 +0000 UTC m=+2154.863746642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" (UID: "55f66ef3-600e-4a16-b61d-f255bb0b943c") : secret "maas-api-e2e-shared-a-w3-a7b7bc-serving-cert" not found
Oct 30 19:40:06.328361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.328331    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.328475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.328382    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.329358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.329331    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.332006 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.331984    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.349779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.349755    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.464473 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:06.464446    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5e966a_d345_47d6_bbd2_997b7fe2d83b.slice/crio-76c072a10e050ff460c93f78fff9530edef85c62719e0a986dc1119654784a3c WatchSource:0}: Error finding container 76c072a10e050ff460c93f78fff9530edef85c62719e0a986dc1119654784a3c: Status 404 returned error can't find the container with id 76c072a10e050ff460c93f78fff9530edef85c62719e0a986dc1119654784a3c
Oct 30 19:40:06.464866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.464728    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:40:06.467823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.467790    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:06.467923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.467853    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:06.467923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.467886    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:06.581710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.581674    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"]
Oct 30 19:40:06.586940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.583885    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4cb13-pg297"]
Oct 30 19:40:06.838045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.837959    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:06.840720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:06.840699    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:07.123037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.122954    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:07.381343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.381239    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerStarted","Data":"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"}
Oct 30 19:40:07.381343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.381314    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerStarted","Data":"76c072a10e050ff460c93f78fff9530edef85c62719e0a986dc1119654784a3c"}
Oct 30 19:40:07.441208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.440885    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podStartSLOduration=2.440865874 podStartE2EDuration="2.440865874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:07.438998028 +0000 UTC m=+2155.474510497" watchObservedRunningTime="2026-10-30 19:40:07.440865874 +0000 UTC m=+2155.476378339"
Oct 30 19:40:07.455427 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.455373    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:40:07.459056 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:07.459009    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f66ef3_600e_4a16_b61d_f255bb0b943c.slice/crio-830f3217fccbaa614c0d4c45fe6125e1db4dbd2c4b6da28b180bab401d3522bf WatchSource:0}: Error finding container 830f3217fccbaa614c0d4c45fe6125e1db4dbd2c4b6da28b180bab401d3522bf: Status 404 returned error can't find the container with id 830f3217fccbaa614c0d4c45fe6125e1db4dbd2c4b6da28b180bab401d3522bf
Oct 30 19:40:07.913062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.913027    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:07.921367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:07.921338    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:08.256919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.256879    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad5c838-bd7f-424b-b2f9-54f5365831b4" path="/var/lib/kubelet/pods/2ad5c838-bd7f-424b-b2f9-54f5365831b4/volumes"
Oct 30 19:40:08.410657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.410271    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" event={"ID":"55f66ef3-600e-4a16-b61d-f255bb0b943c","Type":"ContainerStarted","Data":"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"}
Oct 30 19:40:08.410657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.410338    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" event={"ID":"55f66ef3-600e-4a16-b61d-f255bb0b943c","Type":"ContainerStarted","Data":"830f3217fccbaa614c0d4c45fe6125e1db4dbd2c4b6da28b180bab401d3522bf"}
Oct 30 19:40:08.410657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.410368    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:08.410657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.410466    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:08.411389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.411318    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:40:08.528562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:08.513226    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" podStartSLOduration=2.042949867 podStartE2EDuration="2.51320529s" totalImagesPullingTime="470.255423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:08.455704946 +0000 UTC m=+2156.491217418" watchObservedRunningTime="2026-10-30 19:40:08.51320529 +0000 UTC m=+2156.548717755"
Oct 30 19:40:09.001644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:09.001586    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:40:13.044739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:13.044698    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:40:13.045353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:13.045315    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:14.159837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.159801    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:40:14.174652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.167432    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.185499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.185474    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:40:14.234620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.233529    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-09feeb-dockercfg-7q47n\""
Oct 30 19:40:14.234620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.234494    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-09feeb\""
Oct 30 19:40:14.340286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.340243    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr6t\" (UniqueName: \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.340882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.340854    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.423128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.423034    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:40:14.443362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.443329    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.443551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.443443    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr6t\" (UniqueName: \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.444198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.444175    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.470199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.470153    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr6t\" (UniqueName: \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\") pod \"payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.492419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.492368    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:14.829411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:14.829383    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:40:14.830957 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:14.830925    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9800c198_34fd_4852_b7ab_450765f5a477.slice/crio-7145103cbc760ac3cc4aa0e2c54688f83df604b0b531ef915600e2524b9558a4 WatchSource:0}: Error finding container 7145103cbc760ac3cc4aa0e2c54688f83df604b0b531ef915600e2524b9558a4: Status 404 returned error can't find the container with id 7145103cbc760ac3cc4aa0e2c54688f83df604b0b531ef915600e2524b9558a4
Oct 30 19:40:15.527268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:15.527230    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" event={"ID":"9800c198-34fd-4852-b7ab-450765f5a477","Type":"ContainerStarted","Data":"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"}
Oct 30 19:40:15.527736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:15.527299    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" event={"ID":"9800c198-34fd-4852-b7ab-450765f5a477","Type":"ContainerStarted","Data":"7145103cbc760ac3cc4aa0e2c54688f83df604b0b531ef915600e2524b9558a4"}
Oct 30 19:40:15.527736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:15.527334    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:15.570104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:15.570051    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" podStartSLOduration=1.57003352 podStartE2EDuration="1.57003352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:15.5651952 +0000 UTC m=+2163.600707664" watchObservedRunningTime="2026-10-30 19:40:15.57003352 +0000 UTC m=+2163.605545986"
Oct 30 19:40:18.030322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.030284    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"]
Oct 30 19:40:18.041804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.041770    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822c0b74-5qb77"]
Oct 30 19:40:18.099974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.099941    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab87df39-991c-47cc-a2bd-ec65b76b2267" path="/var/lib/kubelet/pods/ab87df39-991c-47cc-a2bd-ec65b76b2267/volumes"
Oct 30 19:40:18.904332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.904301    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"]
Oct 30 19:40:18.910110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.910089    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:18.925926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.925902    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"]
Oct 30 19:40:18.933439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.933419    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:40:18.933931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:18.933914    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:19.006876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.006845    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.007099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.007072    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffbg\" (UniqueName: \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.007222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.007207    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.108815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.108786    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffbg\" (UniqueName: \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.109262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.108874    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.109262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.109006    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.109697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.109680    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.111357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.111334    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.133184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.133159    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffbg\" (UniqueName: \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\") pod \"maas-api-revoke-sub-0b9e55ba-7pszd\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.224126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.224034    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:19.549201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.549175    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"]
Oct 30 19:40:19.550181 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:19.550151    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7158e90c_3e60_4f65_a1b1_ae115af59943.slice/crio-84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742 WatchSource:0}: Error finding container 84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742: Status 404 returned error can't find the container with id 84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742
Oct 30 19:40:19.560396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:19.560366    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd" event={"ID":"7158e90c-3e60-4f65-a1b1-ae115af59943","Type":"ContainerStarted","Data":"84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742"}
Oct 30 19:40:20.579717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:20.579674    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd" event={"ID":"7158e90c-3e60-4f65-a1b1-ae115af59943","Type":"ContainerStarted","Data":"dac7d6a9ffba124f592d63ad863fc1f6a33c9c8b686263791255e16bb410fdad"}
Oct 30 19:40:20.619930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:20.619879    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd" podStartSLOduration=2.61986157 podStartE2EDuration="2.61986157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:20.618140625 +0000 UTC m=+2168.653653089" watchObservedRunningTime="2026-10-30 19:40:20.61986157 +0000 UTC m=+2168.655374034"
Oct 30 19:40:21.589368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:21.589334    4320 generic.go:356] "Generic (PLEG): container finished" podID="7158e90c-3e60-4f65-a1b1-ae115af59943" containerID="dac7d6a9ffba124f592d63ad863fc1f6a33c9c8b686263791255e16bb410fdad" exitCode=0
Oct 30 19:40:21.589791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:21.589425    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd" event={"ID":"7158e90c-3e60-4f65-a1b1-ae115af59943","Type":"ContainerDied","Data":"dac7d6a9ffba124f592d63ad863fc1f6a33c9c8b686263791255e16bb410fdad"}
Oct 30 19:40:22.794334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.794310    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:22.863383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.863305    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\") pod \"7158e90c-3e60-4f65-a1b1-ae115af59943\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") "
Oct 30 19:40:22.863383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.863370    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\") pod \"7158e90c-3e60-4f65-a1b1-ae115af59943\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") "
Oct 30 19:40:22.863634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.863508    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\" (UniqueName: \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\") pod \"7158e90c-3e60-4f65-a1b1-ae115af59943\" (UID: \"7158e90c-3e60-4f65-a1b1-ae115af59943\") "
Oct 30 19:40:22.863813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.863782    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca" pod "7158e90c-3e60-4f65-a1b1-ae115af59943" (UID: "7158e90c-3e60-4f65-a1b1-ae115af59943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:22.864371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.864345    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:22.865779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.865750    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca" pod "7158e90c-3e60-4f65-a1b1-ae115af59943" (UID: "7158e90c-3e60-4f65-a1b1-ae115af59943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:22.865957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.865931    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg" pod "7158e90c-3e60-4f65-a1b1-ae115af59943" (UID: "7158e90c-3e60-4f65-a1b1-ae115af59943"). InnerVolumeSpecName "kube-api-access-rffbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:22.965394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.965370    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158e90c-3e60-4f65-a1b1-ae115af59943-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:22.965394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:22.965395    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffbg\" (UniqueName: \"kubernetes.io/projected/7158e90c-3e60-4f65-a1b1-ae115af59943-kube-api-access-rffbg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:23.604678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:23.604635    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd" event={"ID":"7158e90c-3e60-4f65-a1b1-ae115af59943","Type":"ContainerDied","Data":"84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742"}
Oct 30 19:40:23.604873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:23.604709    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f0045e382989792a85e94ccc26f98cbc89b0eb0c045a584967ff9f0b2ac742"
Oct 30 19:40:23.604873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:23.604661    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"
Oct 30 19:40:26.538124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:26.538095    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:40:28.646224 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:40:28.646186    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:40:33.904471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.904435    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"]
Oct 30 19:40:33.905156 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.905141    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7158e90c-3e60-4f65-a1b1-ae115af59943" containerName="revoke-keys"
Oct 30 19:40:33.905213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.905158    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7158e90c-3e60-4f65-a1b1-ae115af59943" containerName="revoke-keys"
Oct 30 19:40:33.905278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.905269    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7158e90c-3e60-4f65-a1b1-ae115af59943" containerName="revoke-keys"
Oct 30 19:40:33.910319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.910299    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:33.926932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.926909    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"]
Oct 30 19:40:33.934068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.934036    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:33.934068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:33.934054    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:40:34.029412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.029370    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.029588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.029447    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.029768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.029691    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk24t\" (UniqueName: \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.131308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.131279    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.131446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.131340    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.131516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.131483    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk24t\" (UniqueName: \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.132185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.132152    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.133755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.133728    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.159964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.159895    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk24t\" (UniqueName: \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\") pod \"maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.225189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.225162    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:34.534242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.534217    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"]
Oct 30 19:40:34.536291 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:34.536262    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3a0c5a_d03c_4e2e_91a5_dccf832856a3.slice/crio-53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf WatchSource:0}: Error finding container 53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf: Status 404 returned error can't find the container with id 53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf
Oct 30 19:40:34.721992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:34.717042    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd" event={"ID":"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3","Type":"ContainerStarted","Data":"53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf"}
Oct 30 19:40:35.726540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:35.726503    4320 generic.go:356] "Generic (PLEG): container finished" podID="2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" containerID="79e53a1470706830e3fab7fe74c8205335274deb1cb59f39d2dd81e2962e5d63" exitCode=0
Oct 30 19:40:35.726896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:35.726583    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd" event={"ID":"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3","Type":"ContainerDied","Data":"79e53a1470706830e3fab7fe74c8205335274deb1cb59f39d2dd81e2962e5d63"}
Oct 30 19:40:36.905302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.905270    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:36.976282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.976248    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\" (UniqueName: \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\") pod \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") "
Oct 30 19:40:36.976472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.976375    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\") pod \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") "
Oct 30 19:40:36.976472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.976406    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\") pod \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\" (UID: \"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3\") "
Oct 30 19:40:36.976852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.976821    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca" pod "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" (UID: "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:36.977104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.977082    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:36.978504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.978477    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca" pod "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" (UID: "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:36.979063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:36.979033    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t" pod "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" (UID: "2d3a0c5a-d03c-4e2e-91a5-dccf832856a3"). InnerVolumeSpecName "kube-api-access-nk24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:37.078237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:37.078194    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk24t\" (UniqueName: \"kubernetes.io/projected/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-kube-api-access-nk24t\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:37.078237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:37.078229    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:37.742722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:37.742693    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"
Oct 30 19:40:37.742888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:37.742728    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd" event={"ID":"2d3a0c5a-d03c-4e2e-91a5-dccf832856a3","Type":"ContainerDied","Data":"53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf"}
Oct 30 19:40:37.742888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:37.742787    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53907ec15ad57b1499c6f8d0517b9af5087745c9978cb5e47215d8e136d464cf"
Oct 30 19:40:40.105381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.105348    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"]
Oct 30 19:40:40.106350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.106322    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" containerName="revoke-keys"
Oct 30 19:40:40.106350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.106349    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" containerName="revoke-keys"
Oct 30 19:40:40.106529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.106518    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" containerName="revoke-keys"
Oct 30 19:40:40.110636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.110615    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.129582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.129545    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"]
Oct 30 19:40:40.136480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.136458    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:40.136742 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.136725    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:40:40.222697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.222662    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.222863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.222761    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.223041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.223016    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkl5\" (UniqueName: \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.324173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.324144    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkl5\" (UniqueName: \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.324334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.324317    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.324387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.324365    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.325056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.325032    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.326701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.326680    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.343309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.343280    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkl5\" (UniqueName: \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\") pod \"maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.427511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.427479    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:40.741558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.741531    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"]
Oct 30 19:40:40.742331 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:40.742305    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea98c1bc_edf9_4796_83de_557c7af8fef0.slice/crio-2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94 WatchSource:0}: Error finding container 2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94: Status 404 returned error can't find the container with id 2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94
Oct 30 19:40:40.768662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:40.768628    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr" event={"ID":"ea98c1bc-edf9-4796-83de-557c7af8fef0","Type":"ContainerStarted","Data":"2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94"}
Oct 30 19:40:41.778542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:41.778509    4320 generic.go:356] "Generic (PLEG): container finished" podID="ea98c1bc-edf9-4796-83de-557c7af8fef0" containerID="8f09fef76a04ace716346bdf0b0c24a013a5cec363ab855feb8c0edd9d594377" exitCode=0
Oct 30 19:40:41.778919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:41.778590    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr" event={"ID":"ea98c1bc-edf9-4796-83de-557c7af8fef0","Type":"ContainerDied","Data":"8f09fef76a04ace716346bdf0b0c24a013a5cec363ab855feb8c0edd9d594377"}
Oct 30 19:40:42.942140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.942116    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:42.960690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.960665    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\" (UniqueName: \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\") pod \"ea98c1bc-edf9-4796-83de-557c7af8fef0\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") "
Oct 30 19:40:42.960848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.960836    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\") pod \"ea98c1bc-edf9-4796-83de-557c7af8fef0\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") "
Oct 30 19:40:42.960918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.960906    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\") pod \"ea98c1bc-edf9-4796-83de-557c7af8fef0\" (UID: \"ea98c1bc-edf9-4796-83de-557c7af8fef0\") "
Oct 30 19:40:42.961184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.961160    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca" pod "ea98c1bc-edf9-4796-83de-557c7af8fef0" (UID: "ea98c1bc-edf9-4796-83de-557c7af8fef0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:42.961519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.961502    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:42.962750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.962728    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5" pod "ea98c1bc-edf9-4796-83de-557c7af8fef0" (UID: "ea98c1bc-edf9-4796-83de-557c7af8fef0"). InnerVolumeSpecName "kube-api-access-ddkl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:42.962906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:42.962884    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca" pod "ea98c1bc-edf9-4796-83de-557c7af8fef0" (UID: "ea98c1bc-edf9-4796-83de-557c7af8fef0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:43.062670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:43.062576    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea98c1bc-edf9-4796-83de-557c7af8fef0-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:43.062670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:43.062636    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddkl5\" (UniqueName: \"kubernetes.io/projected/ea98c1bc-edf9-4796-83de-557c7af8fef0-kube-api-access-ddkl5\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:43.793743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:43.793710    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"
Oct 30 19:40:43.793901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:43.793743    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr" event={"ID":"ea98c1bc-edf9-4796-83de-557c7af8fef0","Type":"ContainerDied","Data":"2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94"}
Oct 30 19:40:43.793901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:43.793803    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc3e53fdd474d542c9250dbdcb45ddeaa2fbc1684a92ef89ef2a9ac6e9b6d94"
Oct 30 19:40:46.013872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.013836    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"]
Oct 30 19:40:46.015124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.014735    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ea98c1bc-edf9-4796-83de-557c7af8fef0" containerName="revoke-keys"
Oct 30 19:40:46.015329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.015304    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea98c1bc-edf9-4796-83de-557c7af8fef0" containerName="revoke-keys"
Oct 30 19:40:46.015531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.015517    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea98c1bc-edf9-4796-83de-557c7af8fef0" containerName="revoke-keys"
Oct 30 19:40:46.019578 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.019560    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.041470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.041446    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"]
Oct 30 19:40:46.048249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.048222    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:46.048338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.048271    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:40:46.118897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.118863    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.119054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.119005    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57sn\" (UniqueName: \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.119166 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.119150    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.220781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.220737    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57sn\" (UniqueName: \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.220964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.220890    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.221042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.221014    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.221641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.221617    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.223304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.223278    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.240482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.240462    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57sn\" (UniqueName: \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\") pod \"maas-api-revoke-sub-7d78251b-ncrz2\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.335901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.335814    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:46.651533 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.651493    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"]
Oct 30 19:40:46.652071 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:40:46.652040    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7900f05_1b45_4a47_9072_c19a5b3e52b6.slice/crio-32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d WatchSource:0}: Error finding container 32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d: Status 404 returned error can't find the container with id 32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d
Oct 30 19:40:46.824485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:46.824449    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2" event={"ID":"e7900f05-1b45-4a47-9072-c19a5b3e52b6","Type":"ContainerStarted","Data":"32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d"}
Oct 30 19:40:47.835045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:47.835006    4320 generic.go:356] "Generic (PLEG): container finished" podID="e7900f05-1b45-4a47-9072-c19a5b3e52b6" containerID="3c301db8b06d7096f5144aded92e8a7e07f28bf3ed9d02059b0dc78238e73228" exitCode=0
Oct 30 19:40:47.835508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:47.835101    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2" event={"ID":"e7900f05-1b45-4a47-9072-c19a5b3e52b6","Type":"ContainerDied","Data":"3c301db8b06d7096f5144aded92e8a7e07f28bf3ed9d02059b0dc78238e73228"}
Oct 30 19:40:49.010507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.010481    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:49.085908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.085875    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\") pod \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") "
Oct 30 19:40:49.086104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.085967    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\") pod \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") "
Oct 30 19:40:49.086175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.086123    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\" (UniqueName: \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\") pod \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\" (UID: \"e7900f05-1b45-4a47-9072-c19a5b3e52b6\") "
Oct 30 19:40:49.086334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.086304    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca" pod "e7900f05-1b45-4a47-9072-c19a5b3e52b6" (UID: "e7900f05-1b45-4a47-9072-c19a5b3e52b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:49.087277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.087251    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:49.088774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.088733    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca" pod "e7900f05-1b45-4a47-9072-c19a5b3e52b6" (UID: "e7900f05-1b45-4a47-9072-c19a5b3e52b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:49.089121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.089086    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn" pod "e7900f05-1b45-4a47-9072-c19a5b3e52b6" (UID: "e7900f05-1b45-4a47-9072-c19a5b3e52b6"). InnerVolumeSpecName "kube-api-access-g57sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:49.193007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.192973    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g57sn\" (UniqueName: \"kubernetes.io/projected/e7900f05-1b45-4a47-9072-c19a5b3e52b6-kube-api-access-g57sn\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:49.193007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.193006    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7900f05-1b45-4a47-9072-c19a5b3e52b6-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:40:49.852897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.852867    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"
Oct 30 19:40:49.853080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.852906    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2" event={"ID":"e7900f05-1b45-4a47-9072-c19a5b3e52b6","Type":"ContainerDied","Data":"32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d"}
Oct 30 19:40:49.853080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:49.852965    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b89cf7c8644d9759e3339fcc499df762915d2859971912c2146ba045e1801d"
Oct 30 19:40:51.531623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:40:51.521380    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:40:58.646813 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:40:58.646729    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:41:00.044020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:00.043985    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"]
Oct 30 19:41:00.053098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:00.053067    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4ab4a-4gnqr"]
Oct 30 19:41:00.107404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:00.107371    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea98c1bc-edf9-4796-83de-557c7af8fef0" path="/var/lib/kubelet/pods/ea98c1bc-edf9-4796-83de-557c7af8fef0/volumes"
Oct 30 19:41:07.949710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.949647    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"]
Oct 30 19:41:07.950756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.950722    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e7900f05-1b45-4a47-9072-c19a5b3e52b6" containerName="revoke-keys"
Oct 30 19:41:07.950756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.950752    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7900f05-1b45-4a47-9072-c19a5b3e52b6" containerName="revoke-keys"
Oct 30 19:41:07.951007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.950989    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7900f05-1b45-4a47-9072-c19a5b3e52b6" containerName="revoke-keys"
Oct 30 19:41:07.995060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.995004    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"]
Oct 30 19:41:07.995273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:07.995141    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.011701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.011670    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:08.012049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.012029    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:41:08.079423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.077622    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.079423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.077741    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.079423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.077810    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgtf\" (UniqueName: \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.190516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.190476    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.201801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.193413    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.201970 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.201904    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.202037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.201985    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgtf\" (UniqueName: \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.207644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.206080    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.227279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.227255    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgtf\" (UniqueName: \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\") pod \"maas-api-revoke-sub-301bb9c3-gkrjf\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.276840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.275789    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"]
Oct 30 19:41:08.283682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.283659    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.301276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.301249    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"]
Oct 30 19:41:08.303642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.303616    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.303798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.303771    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2sg\" (UniqueName: \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.303917 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.303870    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.328816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.328791    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:08.407862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.406074    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.407862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.406180    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2sg\" (UniqueName: \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.407862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.407559    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.407862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.407667    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.409517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.409469    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.425196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.425160    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2sg\" (UniqueName: \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\") pod \"maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.603613 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.603528    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:08.640937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.640900    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"]
Oct 30 19:41:08.642242 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:41:08.642212    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23511b6_5c7c_434f_9879_9ee98f3312d7.slice/crio-6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20 WatchSource:0}: Error finding container 6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20: Status 404 returned error can't find the container with id 6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20
Oct 30 19:41:08.910234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:08.910201    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"]
Oct 30 19:41:08.910376 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:41:08.910286    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4d25f4_5ccd_443d_85a8_c6d649b38fc2.slice/crio-ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1 WatchSource:0}: Error finding container ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1: Status 404 returned error can't find the container with id ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1
Oct 30 19:41:09.041302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:09.041271    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf" event={"ID":"a23511b6-5c7c-434f-9879-9ee98f3312d7","Type":"ContainerStarted","Data":"3e629c83e16aa46be071111cfaea21597b124e5f83dc228ff2ef55ad1b5b5db7"}
Oct 30 19:41:09.041722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:09.041327    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf" event={"ID":"a23511b6-5c7c-434f-9879-9ee98f3312d7","Type":"ContainerStarted","Data":"6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20"}
Oct 30 19:41:09.042782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:09.042757    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn" event={"ID":"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2","Type":"ContainerStarted","Data":"ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1"}
Oct 30 19:41:09.093375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:09.093333    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf" podStartSLOduration=2.093319384 podStartE2EDuration="2.093319384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:09.078355246 +0000 UTC m=+2217.113867710" watchObservedRunningTime="2026-10-30 19:41:09.093319384 +0000 UTC m=+2217.128831848"
Oct 30 19:41:10.051485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:10.051446    4320 generic.go:356] "Generic (PLEG): container finished" podID="a23511b6-5c7c-434f-9879-9ee98f3312d7" containerID="3e629c83e16aa46be071111cfaea21597b124e5f83dc228ff2ef55ad1b5b5db7" exitCode=0
Oct 30 19:41:10.051945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:10.051528    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf" event={"ID":"a23511b6-5c7c-434f-9879-9ee98f3312d7","Type":"ContainerDied","Data":"3e629c83e16aa46be071111cfaea21597b124e5f83dc228ff2ef55ad1b5b5db7"}
Oct 30 19:41:10.053405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:10.053378    4320 generic.go:356] "Generic (PLEG): container finished" podID="bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" containerID="57411a186c44a59f5a31b68078944fca7fbebd90c57af1d240458bb359d50049" exitCode=0
Oct 30 19:41:10.053523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:10.053464    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn" event={"ID":"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2","Type":"ContainerDied","Data":"57411a186c44a59f5a31b68078944fca7fbebd90c57af1d240458bb359d50049"}
Oct 30 19:41:11.257438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.257405    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:11.270826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.270789    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:11.348946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.348862    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\" (UniqueName: \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\") pod \"a23511b6-5c7c-434f-9879-9ee98f3312d7\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") "
Oct 30 19:41:11.348946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.348902    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\") pod \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") "
Oct 30 19:41:11.349181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349037    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\" (UniqueName: \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\") pod \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") "
Oct 30 19:41:11.349181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349072    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\") pod \"a23511b6-5c7c-434f-9879-9ee98f3312d7\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") "
Oct 30 19:41:11.349181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349117    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\") pod \"a23511b6-5c7c-434f-9879-9ee98f3312d7\" (UID: \"a23511b6-5c7c-434f-9879-9ee98f3312d7\") "
Oct 30 19:41:11.349484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349461    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\") pod \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\" (UID: \"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2\") "
Oct 30 19:41:11.349836 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349805    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca" pod "a23511b6-5c7c-434f-9879-9ee98f3312d7" (UID: "a23511b6-5c7c-434f-9879-9ee98f3312d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:11.349948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.349863    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca" pod "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" (UID: "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:11.350215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.350197    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:11.350292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.350216    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:11.352210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.351686    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf" pod "a23511b6-5c7c-434f-9879-9ee98f3312d7" (UID: "a23511b6-5c7c-434f-9879-9ee98f3312d7"). InnerVolumeSpecName "kube-api-access-8fgtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:11.352210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.351903    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca" pod "a23511b6-5c7c-434f-9879-9ee98f3312d7" (UID: "a23511b6-5c7c-434f-9879-9ee98f3312d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:11.352658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.352585    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca" pod "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" (UID: "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:11.352658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.352651    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg" pod "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" (UID: "bc4d25f4-5ccd-443d-85a8-c6d649b38fc2"). InnerVolumeSpecName "kube-api-access-zt2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:11.451586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.451558    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgtf\" (UniqueName: \"kubernetes.io/projected/a23511b6-5c7c-434f-9879-9ee98f3312d7-kube-api-access-8fgtf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:11.451586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.451585    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:11.451586 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.451613    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2sg\" (UniqueName: \"kubernetes.io/projected/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2-kube-api-access-zt2sg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:11.451813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:11.451628    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23511b6-5c7c-434f-9879-9ee98f3312d7-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:12.068821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.068795    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"
Oct 30 19:41:12.068994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.068826    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf" event={"ID":"a23511b6-5c7c-434f-9879-9ee98f3312d7","Type":"ContainerDied","Data":"6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20"}
Oct 30 19:41:12.068994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.068875    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c474989b1c95a435e04f22406b60fa0bacd63f0202c62f205e28a15f3d9fc20"
Oct 30 19:41:12.070497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.070472    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"
Oct 30 19:41:12.070497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.070501    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn" event={"ID":"bc4d25f4-5ccd-443d-85a8-c6d649b38fc2","Type":"ContainerDied","Data":"ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1"}
Oct 30 19:41:12.070719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:12.070549    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce012f8e7312a26c54dc2f6a2ddca5b513858cd72948c94070438995e9d516e1"
Oct 30 19:41:13.292860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:13.292821    4320 scope.go:122] "RemoveContainer" containerID="1bb01c5e66ae800e1c80b0719865fa8c7eb50800221deb98a7fa0ed5a8944514"
Oct 30 19:41:13.304288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:13.304264    4320 scope.go:122] "RemoveContainer" containerID="c52ea0b0928940b9c8fd095308bf7516b2e2cbc229e29a6c9df78f0932ece3e0"
Oct 30 19:41:23.779338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.779299    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"]
Oct 30 19:41:23.781097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781065    4320 cpu_manager.go:414] "containerMap: removing container" podUID="bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" containerName="revoke-keys"
Oct 30 19:41:23.781288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781275    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" containerName="revoke-keys"
Oct 30 19:41:23.781439 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781428    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a23511b6-5c7c-434f-9879-9ee98f3312d7" containerName="revoke-keys"
Oct 30 19:41:23.781517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781507    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23511b6-5c7c-434f-9879-9ee98f3312d7" containerName="revoke-keys"
Oct 30 19:41:23.781797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781780    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23511b6-5c7c-434f-9879-9ee98f3312d7" containerName="revoke-keys"
Oct 30 19:41:23.781936 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.781924    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" containerName="revoke-keys"
Oct 30 19:41:23.813622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.788147    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:23.843058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.841150    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:23.851625 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.846185    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"]
Oct 30 19:41:23.940835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.856079    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:41:23.941080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.903830    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:23.941360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.941339    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6lg\" (UniqueName: \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:23.942707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:23.941651    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.049676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.047735    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.049676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.047811    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6lg\" (UniqueName: \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.049676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.047894    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.051153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.051115    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.075664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.065124    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.160658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.160526    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6lg\" (UniqueName: \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\") pod \"maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.229303 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.229278    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:24.807622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:24.806813    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"]
Oct 30 19:41:25.489665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:25.485807    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd" event={"ID":"298a1782-5160-4910-8d7d-c92c058f142c","Type":"ContainerStarted","Data":"6558e4c1b80706fcb8f270f0daa0463ce012a69e9939f30cdccf3f0c12b37248"}
Oct 30 19:41:26.525623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:26.524748    4320 generic.go:356] "Generic (PLEG): container finished" podID="298a1782-5160-4910-8d7d-c92c058f142c" containerID="bb61e0c6e42d79c0ddb3885332399041c661d1887ab0ab113321cc98f0ae584d" exitCode=0
Oct 30 19:41:26.525623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:26.524883    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd" event={"ID":"298a1782-5160-4910-8d7d-c92c058f142c","Type":"ContainerDied","Data":"bb61e0c6e42d79c0ddb3885332399041c661d1887ab0ab113321cc98f0ae584d"}
Oct 30 19:41:27.831095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.831012    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:27.860339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.860311    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\") pod \"298a1782-5160-4910-8d7d-c92c058f142c\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") "
Oct 30 19:41:27.860486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.860465    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\") pod \"298a1782-5160-4910-8d7d-c92c058f142c\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") "
Oct 30 19:41:27.860665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.860651    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\" (UniqueName: \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\") pod \"298a1782-5160-4910-8d7d-c92c058f142c\" (UID: \"298a1782-5160-4910-8d7d-c92c058f142c\") "
Oct 30 19:41:27.860749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.860673    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca" pod "298a1782-5160-4910-8d7d-c92c058f142c" (UID: "298a1782-5160-4910-8d7d-c92c058f142c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:27.861020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.860993    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:27.862665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.862641    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca" pod "298a1782-5160-4910-8d7d-c92c058f142c" (UID: "298a1782-5160-4910-8d7d-c92c058f142c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:27.862960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.862938    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg" pod "298a1782-5160-4910-8d7d-c92c058f142c" (UID: "298a1782-5160-4910-8d7d-c92c058f142c"). InnerVolumeSpecName "kube-api-access-pg6lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:27.962359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.962328    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/298a1782-5160-4910-8d7d-c92c058f142c-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:27.962612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:27.962575    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6lg\" (UniqueName: \"kubernetes.io/projected/298a1782-5160-4910-8d7d-c92c058f142c-kube-api-access-pg6lg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:41:28.540694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:28.540613    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd" event={"ID":"298a1782-5160-4910-8d7d-c92c058f142c","Type":"ContainerDied","Data":"6558e4c1b80706fcb8f270f0daa0463ce012a69e9939f30cdccf3f0c12b37248"}
Oct 30 19:41:28.540929 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:28.540906    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6558e4c1b80706fcb8f270f0daa0463ce012a69e9939f30cdccf3f0c12b37248"
Oct 30 19:41:28.541038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:28.540628    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"
Oct 30 19:41:28.661466 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:41:28.661427    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:41:32.581190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.581150    4320 generic.go:356] "Generic (PLEG): container finished" podID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerID="0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355" exitCode=0
Oct 30 19:41:32.581746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.581220    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerDied","Data":"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"}
Oct 30 19:41:32.581838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.581750    4320 scope.go:122] "RemoveContainer" containerID="0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"
Oct 30 19:41:32.583491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.583465    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:32.583562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.583535    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:32.583637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:32.583569    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:33.589796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.589757    4320 generic.go:356] "Generic (PLEG): container finished" podID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerID="3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4" exitCode=0
Oct 30 19:41:33.590328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.589833    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerDied","Data":"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"}
Oct 30 19:41:33.590434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.590411    4320 scope.go:122] "RemoveContainer" containerID="3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"
Oct 30 19:41:33.592239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.592204    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerStarted","Data":"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"}
Oct 30 19:41:33.592415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.592381    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:33.592469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.592455    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:33.592507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.592494    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:33.913056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.913017    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:41:33.914177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.914146    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:33.914291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:33.914209    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:34.603186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:34.603150    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerStarted","Data":"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"}
Oct 30 19:41:34.914620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:34.914004    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:34.914620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:34.914068    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:35.597621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:35.594494    4320 patch_prober.go:29] interesting pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:41:35.597621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:35.594560    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 19:41:35.597621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:35.594862    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:41:35.915621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:35.914920    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:35.915621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:35.914981    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:36.593620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:36.583646    4320 patch_prober.go:29] interesting pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:41:36.593620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:36.583736    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 19:41:36.914622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:36.913971    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:36.914622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:36.914041    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:37.589741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:37.589700    4320 patch_prober.go:29] interesting pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:41:37.590340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:37.590308    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 19:41:37.916058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:37.916022    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:37.916344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:37.916315    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:38.411724 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:38.411688    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:41:38.590620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:38.589924    4320 patch_prober.go:29] interesting pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:41:38.590620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:38.589983    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 19:41:38.922829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:38.922792    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 19:41:38.923205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:38.923164    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 19:41:39.241353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:39.241274    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:41:39.590670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:39.586948    4320 patch_prober.go:29] interesting pod/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:41:39.590670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:39.587023    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 19:41:39.919624 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:39.919578    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:41:39.920878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:39.920854    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:41:40.642131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:40.642097    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:41:40.644088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:40.644062    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:41:45.795946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:45.795912    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"]
Oct 30 19:41:45.807466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:45.807439    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a7c3b0-2rfpd"]
Oct 30 19:41:46.099872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:46.099791    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3a0c5a-d03c-4e2e-91a5-dccf832856a3" path="/var/lib/kubelet/pods/2d3a0c5a-d03c-4e2e-91a5-dccf832856a3/volumes"
Oct 30 19:41:51.767258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:51.767219    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"]
Oct 30 19:41:51.778989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:51.778961    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e6b67-pqqxn"]
Oct 30 19:41:52.098848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:52.098773    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4d25f4-5ccd-443d-85a8-c6d649b38fc2" path="/var/lib/kubelet/pods/bc4d25f4-5ccd-443d-85a8-c6d649b38fc2/volumes"
Oct 30 19:41:52.443830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:52.443797    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"]
Oct 30 19:41:52.455046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:52.455022    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c8a2a-5c6qd"]
Oct 30 19:41:54.100272 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:41:54.100235    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298a1782-5160-4910-8d7d-c92c058f142c" path="/var/lib/kubelet/pods/298a1782-5160-4910-8d7d-c92c058f142c/volumes"
Oct 30 19:41:58.648522 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:41:58.648474    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:42:04.202024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:04.201282    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:42:13.337065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:13.337035    4320 scope.go:122] "RemoveContainer" containerID="de148771c26792e2e9a0a609abac0990f50f227c90dac7a54cffa8f45a99a9e0"
Oct 30 19:42:13.348828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:13.348794    4320 scope.go:122] "RemoveContainer" containerID="707ea20213cb8cea775b81bfdadf861d8a0282105b72b0e29202aa4038927f7b"
Oct 30 19:42:28.644697 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:42:28.644659    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:42:42.900620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:42.900553    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:42:48.619479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:48.619437    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:42:48.620033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:48.619775    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy" containerID="cri-o://3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374" gracePeriod=30
Oct 30 19:42:48.977224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:48.977184    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:42:48.977438 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:48.977412    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" containerID="cri-o://de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946" gracePeriod=30
Oct 30 19:42:53.169865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:53.169829    4320 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:42:53.170444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:53.170411    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:42:53.930612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:53.930577    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:42:54.038664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.038632    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.038797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.038771    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.038913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.038893    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.038992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.038945    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.039118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039102    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.039188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039141    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.039188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039151    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.039295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039278    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.039365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039348    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039429    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\") pod \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\" (UID: \"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9\") "
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039655    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039773    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.039876    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040178    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040365    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040393    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040409    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040440    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.040506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.040458    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.041576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.041539    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:54.042307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.042277    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "kube-api-access-r7fw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:54.047982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.047953    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:42:54.049105 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.049080    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy" pod "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" (UID: "7f9a8839-d5b5-4929-989a-4e6a19eb6ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.051916 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:42:54.051888    4320 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f4bb69_de05_448b_9657_a91d2f8ce53b.slice/crio-de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f4bb69_de05_448b_9657_a91d2f8ce53b.slice/crio-conmon-de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:42:54.142432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.142391    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.142432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.142421    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.142432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.142431    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fw5\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-kube-api-access-r7fw5\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.142748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.142447    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.253102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.253077    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:42:54.344317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344270    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344338    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344445    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344489    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344565    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344714    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.344834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344807    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.345063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344871    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.345063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.344910    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\") pod \"96f4bb69-de05-448b-9657-a91d2f8ce53b\" (UID: \"96f4bb69-de05-448b-9657-a91d2f8ce53b\") "
Oct 30 19:42:54.346122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.346069    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.346276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.346227    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.346276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.346133    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:54.346465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.346440    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.346682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.346653    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.355242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.355198    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:54.355242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.355230    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:54.355436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.355306    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:42:54.356569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.356540    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f" pod "96f4bb69-de05-448b-9657-a91d2f8ce53b" (UID: "96f4bb69-de05-448b-9657-a91d2f8ce53b"). InnerVolumeSpecName "kube-api-access-9bs4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446826    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446861    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446870    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446878    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bs4f\" (UniqueName: \"kubernetes.io/projected/96f4bb69-de05-448b-9657-a91d2f8ce53b-kube-api-access-9bs4f\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446887    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446897    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446905    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.446911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446913    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f4bb69-de05-448b-9657-a91d2f8ce53b-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.447295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.446922    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f4bb69-de05-448b-9657-a91d2f8ce53b-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.513671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.513640    4320 generic.go:356] "Generic (PLEG): container finished" podID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerID="3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374" exitCode=0
Oct 30 19:42:54.513809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.513723    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"
Oct 30 19:42:54.513809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.513766    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" event={"ID":"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9","Type":"ContainerDied","Data":"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"}
Oct 30 19:42:54.513951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.513829    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt" event={"ID":"7f9a8839-d5b5-4929-989a-4e6a19eb6ca9","Type":"ContainerDied","Data":"2ad02b35f4abf60f05480119ffeab7a8ec7286736611d8f401613726779f145d"}
Oct 30 19:42:54.513951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.513861    4320 scope.go:122] "RemoveContainer" containerID="3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"
Oct 30 19:42:54.515383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.515360    4320 generic.go:356] "Generic (PLEG): container finished" podID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerID="de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946" exitCode=0
Oct 30 19:42:54.515509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.515421    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"
Oct 30 19:42:54.515509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.515452    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" event={"ID":"96f4bb69-de05-448b-9657-a91d2f8ce53b","Type":"ContainerDied","Data":"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"}
Oct 30 19:42:54.515509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.515501    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t" event={"ID":"96f4bb69-de05-448b-9657-a91d2f8ce53b","Type":"ContainerDied","Data":"930795e01ee731d8345b4487308e2b9076aa5f14bdf5dfb712a79cb5e5a1887b"}
Oct 30 19:42:54.525781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.525761    4320 scope.go:122] "RemoveContainer" containerID="3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"
Oct 30 19:42:54.526053 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:42:54.526032    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374\": container with ID starting with 3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374 not found: ID does not exist" containerID="3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"
Oct 30 19:42:54.526102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.526062    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374"} err="failed to get container status \"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374\": rpc error: code = NotFound desc = could not find container \"3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374\": container with ID starting with 3051b4d46840642aff27e838042af74172c28f0a3ac1af78ecc8cd3497e86374 not found: ID does not exist"
Oct 30 19:42:54.526102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.526077    4320 scope.go:122] "RemoveContainer" containerID="de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"
Oct 30 19:42:54.535857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.535834    4320 scope.go:122] "RemoveContainer" containerID="de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"
Oct 30 19:42:54.536166 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:42:54.536148    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946\": container with ID starting with de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946 not found: ID does not exist" containerID="de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"
Oct 30 19:42:54.536233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.536172    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946"} err="failed to get container status \"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946\": rpc error: code = NotFound desc = could not find container \"de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946\": container with ID starting with de884d6e54479b754596f6aed7038a53ac8c0922cb56c88ee7c546c9c9745946 not found: ID does not exist"
Oct 30 19:42:54.570058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.570038    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:42:54.581887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.581864    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-hhtmt"]
Oct 30 19:42:54.611529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.611508    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:42:54.616656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:54.616586    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-hr22t"]
Oct 30 19:42:56.097891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:56.097860    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" path="/var/lib/kubelet/pods/7f9a8839-d5b5-4929-989a-4e6a19eb6ca9/volumes"
Oct 30 19:42:56.098366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:42:56.098351    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" path="/var/lib/kubelet/pods/96f4bb69-de05-448b-9657-a91d2f8ce53b/volumes"
Oct 30 19:42:58.647424 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:42:58.647386    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:43:13.380513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:13.380481    4320 scope.go:122] "RemoveContainer" containerID="7a923cc1f914b6bb36306d0f52eeac42f601ef79333ebf88af925238beedc505"
Oct 30 19:43:14.276899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.276869    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:43:14.277939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.277912    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy"
Oct 30 19:43:14.277939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.277936    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.277960    4320 cpu_manager.go:414] "containerMap: removing container" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.277968    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.277993    4320 cpu_manager.go:414] "containerMap: removing container" podUID="298a1782-5160-4910-8d7d-c92c058f142c" containerName="revoke-keys"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.278001    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298a1782-5160-4910-8d7d-c92c058f142c" containerName="revoke-keys"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.278145    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9a8839-d5b5-4929-989a-4e6a19eb6ca9" containerName="istio-proxy"
Oct 30 19:43:14.278158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.278160    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f4bb69-de05-448b-9657-a91d2f8ce53b" containerName="istio-proxy"
Oct 30 19:43:14.278483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.278176    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="298a1782-5160-4910-8d7d-c92c058f142c" containerName="revoke-keys"
Oct 30 19:43:14.283956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.283934    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.309920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.309884    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:43:14.319668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.317112    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"kube-root-ca.crt\""
Oct 30 19:43:14.319668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.317261    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"openshift-service-ca.crt\""
Oct 30 19:43:14.319668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.317385    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"auth-test-model-d9eb76-kserve-self-signed-certs\""
Oct 30 19:43:14.319668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.317442    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d9eb76\"/\"default-dockercfg-qxdsx\""
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.443986    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b74\" (UniqueName: \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.444055    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.444110    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.444221    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.444533    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.445621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.444588    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546311    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546370    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546478    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b74\" (UniqueName: \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546506    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546568    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.546927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.546901    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.547142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.547117    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.547407 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.547386    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.547468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.547446    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.549509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.549488    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.549706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.549688    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.571728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.571701    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b74\" (UniqueName: \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\") pod \"auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") " pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.609580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.609541    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:14.998079 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:43:14.998045    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60924baf_451e_417c_98b9_3fda824024d2.slice/crio-108e58d377dd4bae369d604fb206ea7a7e76ff4016d450959eaba5fd26dd6fba WatchSource:0}: Error finding container 108e58d377dd4bae369d604fb206ea7a7e76ff4016d450959eaba5fd26dd6fba: Status 404 returned error can't find the container with id 108e58d377dd4bae369d604fb206ea7a7e76ff4016d450959eaba5fd26dd6fba
Oct 30 19:43:14.998746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:14.998716    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:43:15.868547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:15.868510    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" event={"ID":"60924baf-451e-417c-98b9-3fda824024d2","Type":"ContainerStarted","Data":"46478be1918dec4390da6e399ad59181d3775077f7cf283aa9deb850095a4d8c"}
Oct 30 19:43:15.868983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:15.868559    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" event={"ID":"60924baf-451e-417c-98b9-3fda824024d2","Type":"ContainerStarted","Data":"108e58d377dd4bae369d604fb206ea7a7e76ff4016d450959eaba5fd26dd6fba"}
Oct 30 19:43:15.909631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:15.909551    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" podStartSLOduration=1.909537651 podStartE2EDuration="1.909537651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:15.906170077 +0000 UTC m=+2343.941682540" watchObservedRunningTime="2026-10-30 19:43:15.909537651 +0000 UTC m=+2343.945050116"
Oct 30 19:43:16.609682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:16.609654    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:24.609761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:24.609726    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:24.622098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:24.622075    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:24.951758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:24.951729    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:43:28.650677 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:43:28.650639    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:43:35.427402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.427368    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:43:35.432153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.432136    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.447124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.447102    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-614c02-openshift-default-dockercfg-hjlgf\""
Oct 30 19:43:35.487037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.486978    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:43:35.492309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492276    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492382    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492409    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492516    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492550    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492660    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492699    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492818    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.492905 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.492848    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb58z\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.593940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.593904    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.593940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.593945    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594020    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594043    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594108    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594137    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594236    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594265    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb58z\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594348    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594390    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594435    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594534    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594583    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.594906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.594864    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.596530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.596510    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.596682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.596664    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.616358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.616337    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.616728 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.616709    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb58z\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-mdsld\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:35.750031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:35.749959    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:36.095140 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:43:36.095110    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdff107f_517b_4063_8019_4c3d05c65e15.slice/crio-9de12f57f7169e11e14452bcc8642c147030147d0762bb784f7661719aa877d9 WatchSource:0}: Error finding container 9de12f57f7169e11e14452bcc8642c147030147d0762bb784f7661719aa877d9: Status 404 returned error can't find the container with id 9de12f57f7169e11e14452bcc8642c147030147d0762bb784f7661719aa877d9
Oct 30 19:43:36.098646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:36.098608    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:36.098745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:36.098677    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:36.098745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:36.098706    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:36.100415 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:36.100389    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:43:37.036559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.036524    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" event={"ID":"cdff107f-517b-4063-8019-4c3d05c65e15","Type":"ContainerStarted","Data":"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"}
Oct 30 19:43:37.037010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.036587    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" event={"ID":"cdff107f-517b-4063-8019-4c3d05c65e15","Type":"ContainerStarted","Data":"9de12f57f7169e11e14452bcc8642c147030147d0762bb784f7661719aa877d9"}
Oct 30 19:43:37.130645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.130570    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podStartSLOduration=2.130551426 podStartE2EDuration="2.130551426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:37.12876168 +0000 UTC m=+2365.164274124" watchObservedRunningTime="2026-10-30 19:43:37.130551426 +0000 UTC m=+2365.166063891"
Oct 30 19:43:37.750334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.750300    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:37.751532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.751511    4320 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.751627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:37.751555    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 19:43:38.750777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:38.750736    4320 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:43:38.751281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:38.750809    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 19:43:39.750486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:39.750443    4320 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:43:39.750683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:39.750502    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 19:43:40.751190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:40.751148    4320 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:43:40.751626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:40.751207    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 19:43:41.750647 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:41.750580    4320 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:43:41.750822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:41.750663    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 19:43:42.755937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:42.755904    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:42.756427 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:42.756407    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:42.757146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:42.757125    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:43:48.612217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:48.612180    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:43:48.612686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:48.612386    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" containerID="cri-o://400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b" gracePeriod=30
Oct 30 19:43:51.743518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:51.743468    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:43:53.809551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:53.809495    4320 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:43:53.810196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:53.810161    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:43:54.146637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.146573    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:43:54.281683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.281578    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.293696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.284937    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.293696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.286521    4320 generic.go:356] "Generic (PLEG): container finished" podID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerID="400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b" exitCode=0
Oct 30 19:43:54.293934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.286553    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" event={"ID":"862f1849-b8bd-4345-9289-e63b1d26d5fe","Type":"ContainerDied","Data":"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"}
Oct 30 19:43:54.293934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.293791    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s" event={"ID":"862f1849-b8bd-4345-9289-e63b1d26d5fe","Type":"ContainerDied","Data":"9f0741d02413d07170eec3521b7fa1c0e732802a23422f609e53e3f77bb50de8"}
Oct 30 19:43:54.293934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.293826    4320 scope.go:122] "RemoveContainer" containerID="400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"
Oct 30 19:43:54.294099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.286641    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"
Oct 30 19:43:54.294164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.293679    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.294271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.294256    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.295385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.295359    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.298535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.298500    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.300739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.300715    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.301080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.301060    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.307784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.307762    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.308108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.308084    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\") pod \"862f1849-b8bd-4345-9289-e63b1d26d5fe\" (UID: \"862f1849-b8bd-4345-9289-e63b1d26d5fe\") "
Oct 30 19:43:54.310121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.310094    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.316725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.304842    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.316725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.306074    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:54.316725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.316166    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.316725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.316373    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.326116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.326071    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.330562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.330540    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:54.412637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.412121    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420503    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420543    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420560    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420576    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420592    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862f1849-b8bd-4345-9289-e63b1d26d5fe-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420628    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862f1849-b8bd-4345-9289-e63b1d26d5fe-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.434641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.420641    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862f1849-b8bd-4345-9289-e63b1d26d5fe-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.445824 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.444180    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd" pod "862f1849-b8bd-4345-9289-e63b1d26d5fe" (UID: "862f1849-b8bd-4345-9289-e63b1d26d5fe"). InnerVolumeSpecName "kube-api-access-79kjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:54.538948 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.538864    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kjd\" (UniqueName: \"kubernetes.io/projected/862f1849-b8bd-4345-9289-e63b1d26d5fe-kube-api-access-79kjd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.554074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.554043    4320 scope.go:122] "RemoveContainer" containerID="400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"
Oct 30 19:43:54.554628 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:43:54.554556    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b\": container with ID starting with 400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b not found: ID does not exist" containerID="400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"
Oct 30 19:43:54.554768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.554639    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b"} err="failed to get container status \"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b\": rpc error: code = NotFound desc = could not find container \"400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b\": container with ID starting with 400a5a6a51e40da5fd38200a513d9468ba7163af043e4001ad64af41ed5a341b not found: ID does not exist"
Oct 30 19:43:54.676562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.676523    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:43:54.694914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:54.694834    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-j455s"]
Oct 30 19:43:56.098771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:56.098733    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" path="/var/lib/kubelet/pods/862f1849-b8bd-4345-9289-e63b1d26d5fe/volumes"
Oct 30 19:43:58.646720 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:43:58.646684    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:43:58.770425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:58.770388    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:43:58.771077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:43:58.771044    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" containerID="cri-o://fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591" gracePeriod=30
Oct 30 19:44:02.347378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:02.347340    4320 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:44:02.347908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:02.347405    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:44:04.076002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.075974    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188581    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188676    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188768    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188846    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188866    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188880    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.188983    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189006    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189108    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189171    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189233    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\") pod \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\" (UID: \"9c8cfcb8-103d-4119-8eec-2a8dcc31f861\") "
Oct 30 19:44:04.189848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189827    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.189848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.189843    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.190231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.190211    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.190800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.190729    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.191023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.190998    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:04.191118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.191056    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.191345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.191325    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:04.192134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.192110    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "kube-api-access-h2hkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:04.192372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.192355    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo" pod "9c8cfcb8-103d-4119-8eec-2a8dcc31f861" (UID: "9c8cfcb8-103d-4119-8eec-2a8dcc31f861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:04.291463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291429    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291460    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291470    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291477    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291486    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291494    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2hkg\" (UniqueName: \"kubernetes.io/projected/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-kube-api-access-h2hkg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.291712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.291504    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8cfcb8-103d-4119-8eec-2a8dcc31f861-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.376675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.376550    4320 generic.go:356] "Generic (PLEG): container finished" podID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerID="fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591" exitCode=0
Oct 30 19:44:04.376675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.376619    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"
Oct 30 19:44:04.376675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.376623    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" event={"ID":"9c8cfcb8-103d-4119-8eec-2a8dcc31f861","Type":"ContainerDied","Data":"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"}
Oct 30 19:44:04.376907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.376686    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm" event={"ID":"9c8cfcb8-103d-4119-8eec-2a8dcc31f861","Type":"ContainerDied","Data":"7e3212a3b335e62c7cba57c2cd2de2aee7782b7a1c27d6418d548c7e49cfbf32"}
Oct 30 19:44:04.376907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.376720    4320 scope.go:122] "RemoveContainer" containerID="fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"
Oct 30 19:44:04.388428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.388408    4320 scope.go:122] "RemoveContainer" containerID="fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"
Oct 30 19:44:04.388719 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:44:04.388694    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591\": container with ID starting with fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591 not found: ID does not exist" containerID="fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"
Oct 30 19:44:04.388819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.388726    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591"} err="failed to get container status \"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591\": rpc error: code = NotFound desc = could not find container \"fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591\": container with ID starting with fb4541eeb37f6c3afef0e093fe3c26c88dc4716372aa7e52c3b81a006c9e4591 not found: ID does not exist"
Oct 30 19:44:04.429245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.429223    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:44:04.441246 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:04.441211    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-4dfpm"]
Oct 30 19:44:06.098851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:06.098812    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" path="/var/lib/kubelet/pods/9c8cfcb8-103d-4119-8eec-2a8dcc31f861/volumes"
Oct 30 19:44:11.602887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.602847    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 19:44:11.603571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603552    4320 cpu_manager.go:414] "containerMap: removing container" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy"
Oct 30 19:44:11.603571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603571    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy"
Oct 30 19:44:11.603768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603613    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy"
Oct 30 19:44:11.603768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603620    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy"
Oct 30 19:44:11.603867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603772    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="862f1849-b8bd-4345-9289-e63b1d26d5fe" containerName="istio-proxy"
Oct 30 19:44:11.603867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.603783    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8cfcb8-103d-4119-8eec-2a8dcc31f861" containerName="istio-proxy"
Oct 30 19:44:11.607542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.607522    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.651189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.651163    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 19:44:11.685184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685148    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685216    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685465    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685516    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685587    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685671    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685811 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685700    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.685900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.685872    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9msc\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.686039 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.686018    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.786973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.786946    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.786980    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787001    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787023    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787050    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787129    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9msc\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787214    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787250    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787315    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787434    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787458    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787717    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.787940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.787911    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.788162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.788142    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.789781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.789752    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.790100 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.790076    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.806499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.806471    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9msc\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.806739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.806718    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:11.921492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:11.921469    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:12.248348 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.248321    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 19:44:12.249086 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:44:12.249059    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a86e68_8ece_4fee_834e_b8a8c492cbcc.slice/crio-615c5a25a623e173c6840439d5cf33ee91fb96131dce1178b6cd9be096c14690 WatchSource:0}: Error finding container 615c5a25a623e173c6840439d5cf33ee91fb96131dce1178b6cd9be096c14690: Status 404 returned error can't find the container with id 615c5a25a623e173c6840439d5cf33ee91fb96131dce1178b6cd9be096c14690
Oct 30 19:44:12.251909 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.251876    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:12.251990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.251942    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:12.251990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.251977    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:12.450050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.450011    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" event={"ID":"b7a86e68-8ece-4fee-834e-b8a8c492cbcc","Type":"ContainerStarted","Data":"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"}
Oct 30 19:44:12.450221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.450066    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" event={"ID":"b7a86e68-8ece-4fee-834e-b8a8c492cbcc","Type":"ContainerStarted","Data":"615c5a25a623e173c6840439d5cf33ee91fb96131dce1178b6cd9be096c14690"}
Oct 30 19:44:12.495637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.495532    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podStartSLOduration=1.495514291 podStartE2EDuration="1.495514291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:12.49386003 +0000 UTC m=+2400.529372494" watchObservedRunningTime="2026-10-30 19:44:12.495514291 +0000 UTC m=+2400.531026756"
Oct 30 19:44:12.922361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:12.922333    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:13.421226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:13.421194    4320 scope.go:122] "RemoveContainer" containerID="456c01ed981d7a10a237fa266a6119fd90cb6ca399dc5adf1878ee9c2d33bd8f"
Oct 30 19:44:13.431631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:13.431585    4320 scope.go:122] "RemoveContainer" containerID="9b64defd0628bbea4a6a1dfc3e3831c894485547e7384ce8ac9239a48b9a477a"
Oct 30 19:44:13.440916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:13.440895    4320 scope.go:122] "RemoveContainer" containerID="eae06a505fb6c415d64548930e6422326196eaced14ab5d9bd38af26be83b65c"
Oct 30 19:44:13.923851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:13.923820    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:13.924267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:13.923881    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:14.033419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:14.033387    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"]
Oct 30 19:44:14.044953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:14.044921    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bef61f35-jxbb9"]
Oct 30 19:44:14.104475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:14.104434    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f765a77-9e2b-4170-90c7-15c0316b0ccb" path="/var/lib/kubelet/pods/9f765a77-9e2b-4170-90c7-15c0316b0ccb/volumes"
Oct 30 19:44:14.922665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:14.922626    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:14.922854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:14.922705    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:15.922467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:15.922421    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:15.922898 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:15.922487    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:16.922398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:16.922364    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:16.922567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:16.922429    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:17.922681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:17.922646    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:17.923069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:17.922706    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:18.351478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.351396    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"]
Oct 30 19:44:18.357322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.357303    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.397621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.396270    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:18.397621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.396388    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:44:18.414774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.414741    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"]
Oct 30 19:44:18.508492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.508456    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.508492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.508498    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmp4p\" (UniqueName: \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.508742 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.508680    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.610221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.610138    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.610378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.610335    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.610378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.610364    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmp4p\" (UniqueName: \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.610837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.610814    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.612909 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.612883    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.629883 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.629855    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmp4p\" (UniqueName: \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\") pod \"maas-api-revoke-sub-1cce6504-9c5ws\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.678775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.678742    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:18.922686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.922651    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:44:18.922868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.922709    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 19:44:18.988585 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:44:18.988546    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75c2f6c_e0ee_4a11_b1e9_7bb58fa33915.slice/crio-008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf WatchSource:0}: Error finding container 008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf: Status 404 returned error can't find the container with id 008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf
Oct 30 19:44:18.989818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:18.989788    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"]
Oct 30 19:44:19.529180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.529098    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws" event={"ID":"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915","Type":"ContainerStarted","Data":"78dd6071974d1c812550bfaa685265f30740997632c2b3d79a200a8c3909ef68"}
Oct 30 19:44:19.529180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.529163    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws" event={"ID":"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915","Type":"ContainerStarted","Data":"008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf"}
Oct 30 19:44:19.573343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.573288    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws" podStartSLOduration=1.5732714319999999 podStartE2EDuration="1.573271432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:19.57311141 +0000 UTC m=+2407.608623875" watchObservedRunningTime="2026-10-30 19:44:19.573271432 +0000 UTC m=+2407.608783897"
Oct 30 19:44:19.927409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.927382    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:19.927931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.927907    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:19.928807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:19.928789    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 19:44:20.283726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.283641    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:44:20.288295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.288276    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.306489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.306450    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:44:20.355657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.355625    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-08bcfe-openshift-default-dockercfg-hdsw9\""
Oct 30 19:44:20.436832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.436801    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.436832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.436836    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6rd\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.436958    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437142    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437207 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437192    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437294    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437364    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437464    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.437549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.437534    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540366    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540407    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540432    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540484    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540551    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540643    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540695    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540716    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6rd\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.540796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.540782    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.541131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.541004    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.541296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.541269    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.541447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.541426    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.541657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.541555    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.541755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.541654    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.543453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.543427    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.543732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.543703    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.543849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.543817    4320 generic.go:356] "Generic (PLEG): container finished" podID="b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" containerID="78dd6071974d1c812550bfaa685265f30740997632c2b3d79a200a8c3909ef68" exitCode=0
Oct 30 19:44:20.543918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.543895    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws" event={"ID":"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915","Type":"ContainerDied","Data":"78dd6071974d1c812550bfaa685265f30740997632c2b3d79a200a8c3909ef68"}
Oct 30 19:44:20.562329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.562304    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.562420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.562351    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6rd\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.631822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.631798    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:20.968089 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:44:20.968046    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fff4667_95f1_4888_b1ae_fd0c4ba551ac.slice/crio-e659f49a41e53a376e2013d933afcb63fed8e836e17c53a38a2b240ca3d03846 WatchSource:0}: Error finding container e659f49a41e53a376e2013d933afcb63fed8e836e17c53a38a2b240ca3d03846: Status 404 returned error can't find the container with id e659f49a41e53a376e2013d933afcb63fed8e836e17c53a38a2b240ca3d03846
Oct 30 19:44:20.968514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.968221    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:44:20.971275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.971243    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:20.971378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.971309    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:20.971378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:20.971340    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:21.553194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:21.553154    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" event={"ID":"3fff4667-95f1-4888-b1ae-fd0c4ba551ac","Type":"ContainerStarted","Data":"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"}
Oct 30 19:44:21.553410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:21.553219    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" event={"ID":"3fff4667-95f1-4888-b1ae-fd0c4ba551ac","Type":"ContainerStarted","Data":"e659f49a41e53a376e2013d933afcb63fed8e836e17c53a38a2b240ca3d03846"}
Oct 30 19:44:21.612472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:21.612415    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podStartSLOduration=1.612400061 podStartE2EDuration="1.612400061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:21.609051724 +0000 UTC m=+2409.644564213" watchObservedRunningTime="2026-10-30 19:44:21.612400061 +0000 UTC m=+2409.647912525"
Oct 30 19:44:21.632180 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:21.632144    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:22.008197 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.008163    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:22.149345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.149315    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\") pod \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") "
Oct 30 19:44:22.149506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.149373    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\") pod \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") "
Oct 30 19:44:22.149506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.149426    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\" (UniqueName: \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\") pod \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\" (UID: \"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915\") "
Oct 30 19:44:22.149777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.149753    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca" pod "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" (UID: "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:22.149956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.149937    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:22.151591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.151563    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p" pod "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" (UID: "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915"). InnerVolumeSpecName "kube-api-access-qmp4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:22.151591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.151571    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca" pod "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" (UID: "b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:22.252268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.252196    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:22.252268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.252224    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmp4p\" (UniqueName: \"kubernetes.io/projected/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915-kube-api-access-qmp4p\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:22.564404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.564313    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws" event={"ID":"b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915","Type":"ContainerDied","Data":"008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf"}
Oct 30 19:44:22.564404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.564342    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"
Oct 30 19:44:22.564404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.564395    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="008f508fa0f6fb6b3637dbb3a4a94b1bbbd2c8f370a33194fb05320f69d8fadf"
Oct 30 19:44:22.633773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.633740    4320 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:44:22.633933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:22.633803    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:44:23.646619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:23.639035    4320 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:44:23.646619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:23.639097    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:44:24.632958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:24.632929    4320 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.633104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:24.632988    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:44:25.632918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:25.632881    4320 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:44:25.633312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:25.632939    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:44:26.632523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:26.632481    4320 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.632732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:26.632556    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 19:44:27.636924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:27.636894    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:27.637310 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:27.637158    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:27.637890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:27.637872    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:44:28.645145 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:44:28.645111    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:44:29.927188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.927156    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"]
Oct 30 19:44:29.928637 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.928591    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" containerName="revoke-keys"
Oct 30 19:44:29.928777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.928764    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" containerName="revoke-keys"
Oct 30 19:44:29.929110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.929084    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" containerName="revoke-keys"
Oct 30 19:44:29.933001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.932979    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:29.952790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.952765    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"]
Oct 30 19:44:29.956661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.956637    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:29.956772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:29.956700    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:44:30.053387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.053352    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.053568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.053520    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdbr\" (UniqueName: \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.053653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.053637    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.154930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.154888    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdbr\" (UniqueName: \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.155129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.154982    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.155129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.155113    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.155761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.155739    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.157381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.157355    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.174799 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.174776    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdbr\" (UniqueName: \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\") pod \"maas-api-revoke-sub-2d177766-tpcp4\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.257014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.256959    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:30.580612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.580574    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"]
Oct 30 19:44:30.581535 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:44:30.581498    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8597e55_1a5a_4322_8d85_916ad4daf301.slice/crio-0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54 WatchSource:0}: Error finding container 0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54: Status 404 returned error can't find the container with id 0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54
Oct 30 19:44:30.732228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:30.732197    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4" event={"ID":"a8597e55-1a5a-4322-8d85-916ad4daf301","Type":"ContainerStarted","Data":"0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54"}
Oct 30 19:44:31.741468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:31.741390    4320 generic.go:356] "Generic (PLEG): container finished" podID="a8597e55-1a5a-4322-8d85-916ad4daf301" containerID="caa28354eedcdb13df9701de66fc505c65bc48b1032fe1912879e51f97fb3dd8" exitCode=0
Oct 30 19:44:31.741833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:31.741470    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4" event={"ID":"a8597e55-1a5a-4322-8d85-916ad4daf301","Type":"ContainerDied","Data":"caa28354eedcdb13df9701de66fc505c65bc48b1032fe1912879e51f97fb3dd8"}
Oct 30 19:44:32.921743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:32.921711    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:33.012263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.012185    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\") pod \"a8597e55-1a5a-4322-8d85-916ad4daf301\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") "
Oct 30 19:44:33.012416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.012368    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\" (UniqueName: \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\") pod \"a8597e55-1a5a-4322-8d85-916ad4daf301\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") "
Oct 30 19:44:33.012500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.012444    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\") pod \"a8597e55-1a5a-4322-8d85-916ad4daf301\" (UID: \"a8597e55-1a5a-4322-8d85-916ad4daf301\") "
Oct 30 19:44:33.012611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.012551    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca" pod "a8597e55-1a5a-4322-8d85-916ad4daf301" (UID: "a8597e55-1a5a-4322-8d85-916ad4daf301"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:33.013169 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.013143    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:33.014645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.014621    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr" pod "a8597e55-1a5a-4322-8d85-916ad4daf301" (UID: "a8597e55-1a5a-4322-8d85-916ad4daf301"). InnerVolumeSpecName "kube-api-access-jbdbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:33.015042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.015014    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca" pod "a8597e55-1a5a-4322-8d85-916ad4daf301" (UID: "a8597e55-1a5a-4322-8d85-916ad4daf301"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:33.114113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.114088    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbdbr\" (UniqueName: \"kubernetes.io/projected/a8597e55-1a5a-4322-8d85-916ad4daf301-kube-api-access-jbdbr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:33.114113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.114113    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8597e55-1a5a-4322-8d85-916ad4daf301-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:33.759419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.759381    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"
Oct 30 19:44:33.759419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.759401    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4" event={"ID":"a8597e55-1a5a-4322-8d85-916ad4daf301","Type":"ContainerDied","Data":"0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54"}
Oct 30 19:44:33.759752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:33.759451    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7ad9a54b57445afeea12385d5a8b55efd17cbd51982876ac27865d68999b54"
Oct 30 19:44:44.701622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.699639    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"]
Oct 30 19:44:44.719845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.719812    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a8597e55-1a5a-4322-8d85-916ad4daf301" containerName="revoke-keys"
Oct 30 19:44:44.720068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.720053    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8597e55-1a5a-4322-8d85-916ad4daf301" containerName="revoke-keys"
Oct 30 19:44:44.720401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.720384    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8597e55-1a5a-4322-8d85-916ad4daf301" containerName="revoke-keys"
Oct 30 19:44:44.727005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.726982    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.734425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.734136    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"]
Oct 30 19:44:44.747343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.745258    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:44.757445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.757423    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:44:44.826264 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.826230    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.828141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.828112    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.828468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.828448    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjsj\" (UniqueName: \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.930011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.929972    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.930214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.930057    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.930214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.930180    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjsj\" (UniqueName: \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.931140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.931108    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.938265 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.938241    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:44.960810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:44.960737    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjsj\" (UniqueName: \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\") pod \"maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:45.059640 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:45.059564    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:45.378855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:45.378824    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"]
Oct 30 19:44:45.380553 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:44:45.380525    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode654e480_05b3_4c1f_b984_f7997d43beee.slice/crio-1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23 WatchSource:0}: Error finding container 1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23: Status 404 returned error can't find the container with id 1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23
Oct 30 19:44:45.871230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:45.871196    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq" event={"ID":"e654e480-05b3-4c1f-b984-f7997d43beee","Type":"ContainerStarted","Data":"51dd5949428dfac8ad75db532c72824d61bec85538e2dfbb907f2cc28de454e4"}
Oct 30 19:44:45.871667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:45.871259    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq" event={"ID":"e654e480-05b3-4c1f-b984-f7997d43beee","Type":"ContainerStarted","Data":"1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23"}
Oct 30 19:44:45.914670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:45.914613    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq" podStartSLOduration=1.914582941 podStartE2EDuration="1.914582941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:45.911572016 +0000 UTC m=+2433.947084481" watchObservedRunningTime="2026-10-30 19:44:45.914582941 +0000 UTC m=+2433.950095406"
Oct 30 19:44:46.916627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:46.910433    4320 generic.go:356] "Generic (PLEG): container finished" podID="e654e480-05b3-4c1f-b984-f7997d43beee" containerID="51dd5949428dfac8ad75db532c72824d61bec85538e2dfbb907f2cc28de454e4" exitCode=0
Oct 30 19:44:46.916627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:46.910712    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq" event={"ID":"e654e480-05b3-4c1f-b984-f7997d43beee","Type":"ContainerDied","Data":"51dd5949428dfac8ad75db532c72824d61bec85538e2dfbb907f2cc28de454e4"}
Oct 30 19:44:48.101825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.101794    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:48.207134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.207036    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\") pod \"e654e480-05b3-4c1f-b984-f7997d43beee\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") "
Oct 30 19:44:48.207306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.207165    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\" (UniqueName: \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\") pod \"e654e480-05b3-4c1f-b984-f7997d43beee\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") "
Oct 30 19:44:48.207306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.207225    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\") pod \"e654e480-05b3-4c1f-b984-f7997d43beee\" (UID: \"e654e480-05b3-4c1f-b984-f7997d43beee\") "
Oct 30 19:44:48.210828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.208363    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca" pod "e654e480-05b3-4c1f-b984-f7997d43beee" (UID: "e654e480-05b3-4c1f-b984-f7997d43beee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:48.214530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.211486    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj" pod "e654e480-05b3-4c1f-b984-f7997d43beee" (UID: "e654e480-05b3-4c1f-b984-f7997d43beee"). InnerVolumeSpecName "kube-api-access-vfjsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:48.219671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.219640    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca" pod "e654e480-05b3-4c1f-b984-f7997d43beee" (UID: "e654e480-05b3-4c1f-b984-f7997d43beee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:48.318700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.315901    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:48.318700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.315936    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjsj\" (UniqueName: \"kubernetes.io/projected/e654e480-05b3-4c1f-b984-f7997d43beee-kube-api-access-vfjsj\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:48.318700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.315952    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e654e480-05b3-4c1f-b984-f7997d43beee-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:48.936501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.936451    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq" event={"ID":"e654e480-05b3-4c1f-b984-f7997d43beee","Type":"ContainerDied","Data":"1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23"}
Oct 30 19:44:48.936754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.936567    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f039fc9ba927c9b6baa3eb25e492d8d5e1e1f6cd6dec562b55ee0c7e21d0d23"
Oct 30 19:44:48.936754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:48.936452    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"
Oct 30 19:44:49.146721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:49.146638    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:44:49.147161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:49.146846    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy" containerID="cri-o://97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66" gracePeriod=30
Oct 30 19:44:51.327907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:51.327872    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:44:54.443938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.443442    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:44:54.551401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551370    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551424    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551504    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551532    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551641    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551724    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.551835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551746    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:54.551835 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551797    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.552052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551863    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.552052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551902    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:54.552052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.551938    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\") pod \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\" (UID: \"a9bf9b9f-a56e-4da7-8290-622c96e5216e\") "
Oct 30 19:44:54.552052 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552028    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:54.552304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552270    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:54.552387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552363    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:54.552826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552799    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.552951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552831    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.552951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552845    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.552951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552859    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.552951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.552874    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.554377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.554349    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:54.554849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.554817    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "kube-api-access-srs4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:54.554959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.554934    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:54.555376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.555352    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token" pod "a9bf9b9f-a56e-4da7-8290-622c96e5216e" (UID: "a9bf9b9f-a56e-4da7-8290-622c96e5216e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:54.654749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.654633    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.655022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.654997    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.655022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.655022    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srs4v\" (UniqueName: \"kubernetes.io/projected/a9bf9b9f-a56e-4da7-8290-622c96e5216e-kube-api-access-srs4v\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.655194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.655037    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bf9b9f-a56e-4da7-8290-622c96e5216e-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:44:54.986846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.986807    4320 generic.go:356] "Generic (PLEG): container finished" podID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerID="97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66" exitCode=0
Oct 30 19:44:54.987011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.986858    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"
Oct 30 19:44:54.987011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.986885    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" event={"ID":"a9bf9b9f-a56e-4da7-8290-622c96e5216e","Type":"ContainerDied","Data":"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"}
Oct 30 19:44:54.987011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.986937    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v" event={"ID":"a9bf9b9f-a56e-4da7-8290-622c96e5216e","Type":"ContainerDied","Data":"f756227d844a17e8537523c8bb11cbbcf7736ab20d1a24b73316f014031f44c7"}
Oct 30 19:44:54.987011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:54.986962    4320 scope.go:122] "RemoveContainer" containerID="97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"
Oct 30 19:44:55.001660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:55.001641    4320 scope.go:122] "RemoveContainer" containerID="97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"
Oct 30 19:44:55.001925 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:44:55.001906    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66\": container with ID starting with 97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66 not found: ID does not exist" containerID="97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"
Oct 30 19:44:55.001971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:55.001932    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66"} err="failed to get container status \"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66\": rpc error: code = NotFound desc = could not find container \"97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66\": container with ID starting with 97155d135db2e1795d471a81d6c0c5f46de2cba6736c5f12021958af4b3c7c66 not found: ID does not exist"
Oct 30 19:44:55.098212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:55.098183    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:44:55.113402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:55.113373    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dvd8v"]
Oct 30 19:44:56.102068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:44:56.102030    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" path="/var/lib/kubelet/pods/a9bf9b9f-a56e-4da7-8290-622c96e5216e/volumes"
Oct 30 19:44:58.644795 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:44:58.644753    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:45:00.189449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.189417    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:00.190473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190453    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e654e480-05b3-4c1f-b984-f7997d43beee" containerName="revoke-keys"
Oct 30 19:45:00.190520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190476    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e654e480-05b3-4c1f-b984-f7997d43beee" containerName="revoke-keys"
Oct 30 19:45:00.190556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190536    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy"
Oct 30 19:45:00.190556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190545    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy"
Oct 30 19:45:00.190714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190701    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e654e480-05b3-4c1f-b984-f7997d43beee" containerName="revoke-keys"
Oct 30 19:45:00.190755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.190728    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bf9b9f-a56e-4da7-8290-622c96e5216e" containerName="istio-proxy"
Oct 30 19:45:00.196453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.196430    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:00.196568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.196543    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.200708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.200688    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.209012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.208992    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:00.213962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.213947    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.220046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.220025    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:00.227721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.226909    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:45:00.227721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.227108    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:00.250366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.250340    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:00.257213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.256703    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.257350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.257277    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:00.263990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.263963    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:00.264267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.264194    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.270900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.270871    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:00.271012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.270997    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.314634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.305763    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:00.320919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.320896    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:00.331709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.331167    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:00.331709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.331284    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.333726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.333701    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xld59\" (UniqueName: \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.333855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.333769    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.333855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.333824    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.333973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.333861    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hz4\" (UniqueName: \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.333973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.333941    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrhr\" (UniqueName: \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.334084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334010    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334141    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6tv\" (UniqueName: \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334183    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334209    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz85g\" (UniqueName: \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334265    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x526b\" (UniqueName: \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334346    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.334446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.334375    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.336334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.336313    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:00.344686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.344668    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:00.345756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.344878    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.355525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.355506    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:00.355829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.355759    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.358299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.357992    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:00.364031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.364011    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.366692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.366675    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:00.371950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.371636    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:00.376037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.376009    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:00.381215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.381164    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:00.385927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.385912    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.387720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.386835    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:00.402381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.402356    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.408058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.406319    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:00.409874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.409847    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:00.414241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.414219    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:00.435661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435632    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6tv\" (UniqueName: \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.435768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435671    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.435768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435691    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz85g\" (UniqueName: \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.435768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435743    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x526b\" (UniqueName: \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.435954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzf5\" (UniqueName: \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.435954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435847    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.435954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.435936    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.436129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436058    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.436129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436105    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.436243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436179    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xld59\" (UniqueName: \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.436371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436209    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5nz\" (UniqueName: \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.436442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436430    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.436503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436460    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.436503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436460    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.436620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436524    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.436620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436566    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hz4\" (UniqueName: \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.436739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436684    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.436828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436806    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.436828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436821    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrhr\" (UniqueName: \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.436932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436873    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8ph\" (UniqueName: \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.436988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.436956    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.437055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.437022    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.437219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.437185    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgvm\" (UniqueName: \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.437342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.437324    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.437660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.437640    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.438162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.438142    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.467780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.467716    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hz4\" (UniqueName: \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.467987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.467874    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz85g\" (UniqueName: \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\") pod \"maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.470031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.470009    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6tv\" (UniqueName: \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\") pod \"maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.470331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.470310    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xld59\" (UniqueName: \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.470377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.470346    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x526b\" (UniqueName: \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.471306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.471287    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrhr\" (UniqueName: \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\") pod \"maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.514356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.514336    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:00.524862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.524842    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:00.536788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.536763    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:00.539074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.538699    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5nz\" (UniqueName: \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.539295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539270    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.539675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539495    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8ph\" (UniqueName: \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.539675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539565    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r579n\" (UniqueName: \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.539675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539630    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.539864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539728    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgvm\" (UniqueName: \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.539864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539784    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5dn\" (UniqueName: \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.539952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539877    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzf5\" (UniqueName: \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.540002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539948    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.540002 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.539984    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.540073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.540012    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.540073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.540054    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.541356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.541303    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.542086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.542053    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.542245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.542174    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.542245 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.542208    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.565611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.564834    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5nz\" (UniqueName: \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\") pod \"maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.575288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.574033    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgvm\" (UniqueName: \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.576152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.576100    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8ph\" (UniqueName: \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\") pod \"maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.577139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.577114    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzf5\" (UniqueName: \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.613021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.609709    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:00.630820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.630733    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:00.635690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.635662    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:00.645662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.643578    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5dn\" (UniqueName: \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.645662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.643738    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.645662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.643775    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.645662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.643976    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r579n\" (UniqueName: \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.645662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.645587    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.651102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.648838    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.651102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.648999    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:00.667147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.667069    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r579n\" (UniqueName: \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.668658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.668481    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:00.674333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.673498    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5dn\" (UniqueName: \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\") pod \"maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.682985 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.682832    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:00.704070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.703860    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:00.718519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.718284    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:00.728412 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.728189    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:00.860152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:00.860111    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:00.861658 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:00.861625    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f267b1c_3898_4d33_a0a4_f64b5fb9e769.slice/crio-f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4 WatchSource:0}: Error finding container f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4: Status 404 returned error can't find the container with id f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4
Oct 30 19:45:01.057187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.057127    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerStarted","Data":"f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4"}
Oct 30 19:45:01.062340 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.062314    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:01.068994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.068942    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"]
Oct 30 19:45:01.074817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.074794    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608d9fce-4fbmg"]
Oct 30 19:45:01.080392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.080370    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:01.177250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.177227    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:01.182243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.182215    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:01.260426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.260398    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:01.260890 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.260516    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:01.265024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.264994    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:01.273647 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.273233    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b3e6d_4189_4984_a862_89fb78fdd12c.slice/crio-2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea WatchSource:0}: Error finding container 2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea: Status 404 returned error can't find the container with id 2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea
Oct 30 19:45:01.274008 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.273965    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4 WatchSource:0}: Error finding container 97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4: Status 404 returned error can't find the container with id 97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4
Oct 30 19:45:01.279259 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.279242    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde68579_b9aa_43d6_a5fb_f60718c906a1.slice/crio-338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23 WatchSource:0}: Error finding container 338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23: Status 404 returned error can't find the container with id 338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23
Oct 30 19:45:01.279744 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.279724    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993 WatchSource:0}: Error finding container 1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993: Status 404 returned error can't find the container with id 1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993
Oct 30 19:45:01.284382 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.284359    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b WatchSource:0}: Error finding container 9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b: Status 404 returned error can't find the container with id 9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b
Oct 30 19:45:01.320021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.319998    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:01.324081 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.324053    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00 WatchSource:0}: Error finding container fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00: Status 404 returned error can't find the container with id fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00
Oct 30 19:45:01.347889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.347864    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:01.350583 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.350561    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c WatchSource:0}: Error finding container c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c: Status 404 returned error can't find the container with id c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c
Oct 30 19:45:01.358123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.358095    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:01.363065 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.363039    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49943423_8173_4ac7_bb6d_c0324d84d6dd.slice/crio-10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb WatchSource:0}: Error finding container 10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb: Status 404 returned error can't find the container with id 10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb
Oct 30 19:45:01.378317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:01.378293    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:01.477752 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:01.477723    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9886b8b7_4bd0_487b_af82_94b47fc2fd39.slice/crio-64ce60e2962b61a006ed4add1c3f2c7b47066501582c0fdd0b25ffd3728ed122 WatchSource:0}: Error finding container 64ce60e2962b61a006ed4add1c3f2c7b47066501582c0fdd0b25ffd3728ed122: Status 404 returned error can't find the container with id 64ce60e2962b61a006ed4add1c3f2c7b47066501582c0fdd0b25ffd3728ed122
Oct 30 19:45:02.066341 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.066308    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" event={"ID":"49943423-8173-4ac7-bb6d-c0324d84d6dd","Type":"ContainerStarted","Data":"10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb"}
Oct 30 19:45:02.067726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.067702    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerStarted","Data":"9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b"}
Oct 30 19:45:02.072427 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.072339    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" event={"ID":"fde68579-b9aa-43d6-a5fb-f60718c906a1","Type":"ContainerStarted","Data":"338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23"}
Oct 30 19:45:02.074841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.074804    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerStarted","Data":"9c432c03de74e1a23149e6abeca3ab2b2f78e2d8867c45a01e91987d9da88731"}
Oct 30 19:45:02.074943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.074853    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerStarted","Data":"fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00"}
Oct 30 19:45:02.077456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.077434    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerStarted","Data":"666d7c1fb7cd55fb666ba3bd821dd31dbf9f60f44406adbb0dfc3c3919935716"}
Oct 30 19:45:02.077544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.077475    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerStarted","Data":"85b6be531bb9703c4e176b0b67b168985c268065e5bb9f67ab4a8db0f1e68a35"}
Oct 30 19:45:02.078996 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.078975    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerStarted","Data":"c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c"}
Oct 30 19:45:02.080580 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.080556    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" event={"ID":"df18a818-7dd3-4ddc-a2e1-300af1320e2a","Type":"ContainerStarted","Data":"97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4"}
Oct 30 19:45:02.082061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.082039    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerStarted","Data":"1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993"}
Oct 30 19:45:02.083920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.083895    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerStarted","Data":"884f7020dbd9a63a8bbf81894d84ce3026774beeda8f481bb67f8877364bc204"}
Oct 30 19:45:02.084025 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.083935    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerStarted","Data":"f00c866d31df8eb3a26a1379759bf67c8439d32de3a155aecbaa3f0ab2f989de"}
Oct 30 19:45:02.085426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.085404    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" event={"ID":"9886b8b7-4bd0-487b-af82-94b47fc2fd39","Type":"ContainerStarted","Data":"64ce60e2962b61a006ed4add1c3f2c7b47066501582c0fdd0b25ffd3728ed122"}
Oct 30 19:45:02.087740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.087707    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerStarted","Data":"5c84e65c36fca3140237eeb111ebabded136008b2dde84f1996ff86d08fd680c"}
Oct 30 19:45:02.087832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.087752    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerStarted","Data":"2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea"}
Oct 30 19:45:02.100699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.100675    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4c4c1d-7304-47ed-8394-67ed6908942d" path="/var/lib/kubelet/pods/dc4c4c1d-7304-47ed-8394-67ed6908942d/volumes"
Oct 30 19:45:02.101041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.101025    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerStarted","Data":"3b82ace67bc24204ef1434e3142a0ef0d6fa2c0b2cf6175c1d14465eeffa31ac"}
Oct 30 19:45:02.183709 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.183654    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" podStartSLOduration=2.183583573 podStartE2EDuration="2.183583573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.180781783 +0000 UTC m=+2450.216294249" watchObservedRunningTime="2026-10-30 19:45:02.183583573 +0000 UTC m=+2450.219096036"
Oct 30 19:45:02.183863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.183765    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" podStartSLOduration=2.183758011 podStartE2EDuration="2.183758011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.127950857 +0000 UTC m=+2450.163463320" watchObservedRunningTime="2026-10-30 19:45:02.183758011 +0000 UTC m=+2450.219270475"
Oct 30 19:45:02.228091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.228044    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" podStartSLOduration=2.228029857 podStartE2EDuration="2.228029857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.224584716 +0000 UTC m=+2450.260097181" watchObservedRunningTime="2026-10-30 19:45:02.228029857 +0000 UTC m=+2450.263542322"
Oct 30 19:45:02.266667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.266615    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" podStartSLOduration=2.266584532 podStartE2EDuration="2.266584532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.262487725 +0000 UTC m=+2450.298000188" watchObservedRunningTime="2026-10-30 19:45:02.266584532 +0000 UTC m=+2450.302096997"
Oct 30 19:45:02.301182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:02.301135    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" podStartSLOduration=2.301123215 podStartE2EDuration="2.301123215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.299409936 +0000 UTC m=+2450.334922400" watchObservedRunningTime="2026-10-30 19:45:02.301123215 +0000 UTC m=+2450.336635679"
Oct 30 19:45:03.104056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.104029    4320 generic.go:356] "Generic (PLEG): container finished" podID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerID="fd45f460f6f11c9b995a3b6e5f92d05fc349105efebbb5b96187c0c20c2a3ea3" exitCode=60
Oct 30 19:45:03.104181 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.104111    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerDied","Data":"fd45f460f6f11c9b995a3b6e5f92d05fc349105efebbb5b96187c0c20c2a3ea3"}
Oct 30 19:45:03.104377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.104361    4320 scope.go:122] "RemoveContainer" containerID="fd45f460f6f11c9b995a3b6e5f92d05fc349105efebbb5b96187c0c20c2a3ea3"
Oct 30 19:45:03.105966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.105939    4320 generic.go:356] "Generic (PLEG): container finished" podID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerID="884f7020dbd9a63a8bbf81894d84ce3026774beeda8f481bb67f8877364bc204" exitCode=60
Oct 30 19:45:03.106066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.106048    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerDied","Data":"884f7020dbd9a63a8bbf81894d84ce3026774beeda8f481bb67f8877364bc204"}
Oct 30 19:45:03.106282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.106263    4320 scope.go:122] "RemoveContainer" containerID="884f7020dbd9a63a8bbf81894d84ce3026774beeda8f481bb67f8877364bc204"
Oct 30 19:45:03.108201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.108184    4320 generic.go:356] "Generic (PLEG): container finished" podID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerID="6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615" exitCode=60
Oct 30 19:45:03.108251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.108240    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" event={"ID":"9886b8b7-4bd0-487b-af82-94b47fc2fd39","Type":"ContainerDied","Data":"6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"}
Oct 30 19:45:03.108514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.108498    4320 scope.go:122] "RemoveContainer" containerID="6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"
Oct 30 19:45:03.110656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.110637    4320 generic.go:356] "Generic (PLEG): container finished" podID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerID="5c84e65c36fca3140237eeb111ebabded136008b2dde84f1996ff86d08fd680c" exitCode=60
Oct 30 19:45:03.110731 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.110703    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerDied","Data":"5c84e65c36fca3140237eeb111ebabded136008b2dde84f1996ff86d08fd680c"}
Oct 30 19:45:03.110920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.110905    4320 scope.go:122] "RemoveContainer" containerID="5c84e65c36fca3140237eeb111ebabded136008b2dde84f1996ff86d08fd680c"
Oct 30 19:45:03.112403 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.112382    4320 generic.go:356] "Generic (PLEG): container finished" podID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerID="3b82ace67bc24204ef1434e3142a0ef0d6fa2c0b2cf6175c1d14465eeffa31ac" exitCode=60
Oct 30 19:45:03.112487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.112465    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerDied","Data":"3b82ace67bc24204ef1434e3142a0ef0d6fa2c0b2cf6175c1d14465eeffa31ac"}
Oct 30 19:45:03.112719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.112702    4320 scope.go:122] "RemoveContainer" containerID="3b82ace67bc24204ef1434e3142a0ef0d6fa2c0b2cf6175c1d14465eeffa31ac"
Oct 30 19:45:03.114225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.114205    4320 generic.go:356] "Generic (PLEG): container finished" podID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerID="2f7aa7a918d2e2830cae87f85bead552ce7357f0764a6b40c4dbfd9c5e79d53e" exitCode=60
Oct 30 19:45:03.114316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.114225    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" event={"ID":"49943423-8173-4ac7-bb6d-c0324d84d6dd","Type":"ContainerDied","Data":"2f7aa7a918d2e2830cae87f85bead552ce7357f0764a6b40c4dbfd9c5e79d53e"}
Oct 30 19:45:03.114509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.114486    4320 scope.go:122] "RemoveContainer" containerID="2f7aa7a918d2e2830cae87f85bead552ce7357f0764a6b40c4dbfd9c5e79d53e"
Oct 30 19:45:03.115944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.115924    4320 generic.go:356] "Generic (PLEG): container finished" podID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerID="4dda27fcbdff3d5436c7e326163dba72adf7116ffebf51fa182de4b7a25bb478" exitCode=60
Oct 30 19:45:03.116314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.116290    4320 scope.go:122] "RemoveContainer" containerID="4dda27fcbdff3d5436c7e326163dba72adf7116ffebf51fa182de4b7a25bb478"
Oct 30 19:45:03.116563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.116510    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerDied","Data":"4dda27fcbdff3d5436c7e326163dba72adf7116ffebf51fa182de4b7a25bb478"}
Oct 30 19:45:03.118448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.118429    4320 generic.go:356] "Generic (PLEG): container finished" podID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerID="b9725ffb234c13920c61fb10e9c5c4452d18a8f4c43f7d5058ad029df74a2fc5" exitCode=60
Oct 30 19:45:03.118535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.118509    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" event={"ID":"fde68579-b9aa-43d6-a5fb-f60718c906a1","Type":"ContainerDied","Data":"b9725ffb234c13920c61fb10e9c5c4452d18a8f4c43f7d5058ad029df74a2fc5"}
Oct 30 19:45:03.118735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.118715    4320 scope.go:122] "RemoveContainer" containerID="b9725ffb234c13920c61fb10e9c5c4452d18a8f4c43f7d5058ad029df74a2fc5"
Oct 30 19:45:03.121679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.120936    4320 generic.go:356] "Generic (PLEG): container finished" podID="78f0bf84-4d74-451f-b984-36431daa554b" containerID="9c432c03de74e1a23149e6abeca3ab2b2f78e2d8867c45a01e91987d9da88731" exitCode=60
Oct 30 19:45:03.121679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.121612    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerDied","Data":"9c432c03de74e1a23149e6abeca3ab2b2f78e2d8867c45a01e91987d9da88731"}
Oct 30 19:45:03.122301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.122115    4320 scope.go:122] "RemoveContainer" containerID="9c432c03de74e1a23149e6abeca3ab2b2f78e2d8867c45a01e91987d9da88731"
Oct 30 19:45:03.129177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.128792    4320 generic.go:356] "Generic (PLEG): container finished" podID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerID="666d7c1fb7cd55fb666ba3bd821dd31dbf9f60f44406adbb0dfc3c3919935716" exitCode=60
Oct 30 19:45:03.129177 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.128866    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerDied","Data":"666d7c1fb7cd55fb666ba3bd821dd31dbf9f60f44406adbb0dfc3c3919935716"}
Oct 30 19:45:03.130051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.129332    4320 scope.go:122] "RemoveContainer" containerID="666d7c1fb7cd55fb666ba3bd821dd31dbf9f60f44406adbb0dfc3c3919935716"
Oct 30 19:45:03.135017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.134988    4320 generic.go:356] "Generic (PLEG): container finished" podID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerID="d542f68be26b9f8a930bc343bd062d053239a6aeef1074a525669d3fe274c7df" exitCode=60
Oct 30 19:45:03.135131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.135099    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerDied","Data":"d542f68be26b9f8a930bc343bd062d053239a6aeef1074a525669d3fe274c7df"}
Oct 30 19:45:03.135342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.135320    4320 scope.go:122] "RemoveContainer" containerID="d542f68be26b9f8a930bc343bd062d053239a6aeef1074a525669d3fe274c7df"
Oct 30 19:45:03.136979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.136956    4320 generic.go:356] "Generic (PLEG): container finished" podID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerID="c7a2fb3c7a6195dbd411a186be77fc47966bd8a50d545705abeb469dbf6991b0" exitCode=60
Oct 30 19:45:03.137059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.137033    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" event={"ID":"df18a818-7dd3-4ddc-a2e1-300af1320e2a","Type":"ContainerDied","Data":"c7a2fb3c7a6195dbd411a186be77fc47966bd8a50d545705abeb469dbf6991b0"}
Oct 30 19:45:03.137292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.137271    4320 scope.go:122] "RemoveContainer" containerID="c7a2fb3c7a6195dbd411a186be77fc47966bd8a50d545705abeb469dbf6991b0"
Oct 30 19:45:03.310315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:03.310261    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:04.148226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.148189    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerStarted","Data":"cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"}
Oct 30 19:45:04.150301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.150268    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerStarted","Data":"851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"}
Oct 30 19:45:04.152275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.152250    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" event={"ID":"9886b8b7-4bd0-487b-af82-94b47fc2fd39","Type":"ContainerStarted","Data":"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"}
Oct 30 19:45:04.152372 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.152289    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup" containerID="cri-o://720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20" gracePeriod=30
Oct 30 19:45:04.154370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.154346    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerStarted","Data":"f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"}
Oct 30 19:45:04.156300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.156274    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerStarted","Data":"cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"}
Oct 30 19:45:04.158335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.158311    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerStarted","Data":"f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"}
Oct 30 19:45:04.160430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.160409    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerStarted","Data":"a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"}
Oct 30 19:45:04.194677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.194652    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"]
Oct 30 19:45:04.198491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.198449    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" podStartSLOduration=4.198436238 podStartE2EDuration="4.198436238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.191108025 +0000 UTC m=+2452.226620491" watchObservedRunningTime="2026-10-30 19:45:04.198436238 +0000 UTC m=+2452.233948701"
Oct 30 19:45:04.203129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.203108    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5d92fa-rf7xq"]
Oct 30 19:45:04.242091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.242027    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" podStartSLOduration=4.242007243 podStartE2EDuration="4.242007243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.240685592 +0000 UTC m=+2452.276198057" watchObservedRunningTime="2026-10-30 19:45:04.242007243 +0000 UTC m=+2452.277519711"
Oct 30 19:45:04.321477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.321428    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" podStartSLOduration=4.321414579 podStartE2EDuration="4.321414579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.319399883 +0000 UTC m=+2452.354912348" watchObservedRunningTime="2026-10-30 19:45:04.321414579 +0000 UTC m=+2452.356927042"
Oct 30 19:45:04.421884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:04.421835    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" podStartSLOduration=4.421822323 podStartE2EDuration="4.421822323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.416916181 +0000 UTC m=+2452.452428647" watchObservedRunningTime="2026-10-30 19:45:04.421822323 +0000 UTC m=+2452.457334786"
Oct 30 19:45:05.176398 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.176368    4320 generic.go:356] "Generic (PLEG): container finished" podID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerID="c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580" exitCode=60
Oct 30 19:45:05.176535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.176424    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" event={"ID":"df18a818-7dd3-4ddc-a2e1-300af1320e2a","Type":"ContainerDied","Data":"c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580"}
Oct 30 19:45:05.176535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.176497    4320 scope.go:122] "RemoveContainer" containerID="c7a2fb3c7a6195dbd411a186be77fc47966bd8a50d545705abeb469dbf6991b0"
Oct 30 19:45:05.176841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.176817    4320 scope.go:122] "RemoveContainer" containerID="c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580"
Oct 30 19:45:05.177238 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.177208    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9_odh-ai-gateway-infra(df18a818-7dd3-4ddc-a2e1-300af1320e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a"
Oct 30 19:45:05.179005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.178985    4320 generic.go:356] "Generic (PLEG): container finished" podID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerID="cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542" exitCode=60
Oct 30 19:45:05.179095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.179070    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerDied","Data":"cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"}
Oct 30 19:45:05.179319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.179300    4320 scope.go:122] "RemoveContainer" containerID="cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"
Oct 30 19:45:05.179678 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.179657    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9_odh-ai-gateway-infra(2f4732d9-f4ab-4d7d-a5ad-49c97692c41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b"
Oct 30 19:45:05.181126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.181103    4320 generic.go:356] "Generic (PLEG): container finished" podID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerID="851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225" exitCode=60
Oct 30 19:45:05.181223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.181205    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerDied","Data":"851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"}
Oct 30 19:45:05.181462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.181447    4320 scope.go:122] "RemoveContainer" containerID="851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"
Oct 30 19:45:05.181805 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.181784    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c_odh-ai-gateway-infra(ca784aee-fc45-4f3f-8300-06c9b686f604)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604"
Oct 30 19:45:05.183417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.183392    4320 generic.go:356] "Generic (PLEG): container finished" podID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerID="f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff" exitCode=60
Oct 30 19:45:05.183495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.183432    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerDied","Data":"f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"}
Oct 30 19:45:05.183696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.183680    4320 scope.go:122] "RemoveContainer" containerID="f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"
Oct 30 19:45:05.183945 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.183921    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7_odh-ai-gateway-infra(b61b3e6d-4189-4984-a862-89fb78fdd12c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c"
Oct 30 19:45:05.186332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.186300    4320 generic.go:356] "Generic (PLEG): container finished" podID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerID="5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a" exitCode=60
Oct 30 19:45:05.186447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.186330    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerDied","Data":"5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a"}
Oct 30 19:45:05.186670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.186653    4320 scope.go:122] "RemoveContainer" containerID="5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a"
Oct 30 19:45:05.186948 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.186928    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv_odh-ai-gateway-infra(8f267b1c-3898-4d33-a0a4-f64b5fb9e769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769"
Oct 30 19:45:05.188681 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.188658    4320 generic.go:356] "Generic (PLEG): container finished" podID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerID="f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1" exitCode=60
Oct 30 19:45:05.188774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.188735    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" event={"ID":"49943423-8173-4ac7-bb6d-c0324d84d6dd","Type":"ContainerDied","Data":"f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1"}
Oct 30 19:45:05.188995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.188978    4320 scope.go:122] "RemoveContainer" containerID="f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1"
Oct 30 19:45:05.189271 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.189247    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m_odh-ai-gateway-infra(49943423-8173-4ac7-bb6d-c0324d84d6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd"
Oct 30 19:45:05.192566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.192545    4320 generic.go:356] "Generic (PLEG): container finished" podID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerID="cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a" exitCode=60
Oct 30 19:45:05.192764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.192743    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerDied","Data":"cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"}
Oct 30 19:45:05.193026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.193007    4320 scope.go:122] "RemoveContainer" containerID="cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"
Oct 30 19:45:05.193334 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.193305    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr_odh-ai-gateway-infra(88d6df1a-d2ca-4856-9535-7405a748f19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b"
Oct 30 19:45:05.194944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.194916    4320 generic.go:356] "Generic (PLEG): container finished" podID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerID="3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f" exitCode=60
Oct 30 19:45:05.195018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.194997    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" event={"ID":"fde68579-b9aa-43d6-a5fb-f60718c906a1","Type":"ContainerDied","Data":"3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f"}
Oct 30 19:45:05.195234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.195215    4320 scope.go:122] "RemoveContainer" containerID="3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f"
Oct 30 19:45:05.195344 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.195258    4320 scope.go:122] "RemoveContainer" containerID="fd45f460f6f11c9b995a3b6e5f92d05fc349105efebbb5b96187c0c20c2a3ea3"
Oct 30 19:45:05.195519 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.195497    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr_odh-ai-gateway-infra(fde68579-b9aa-43d6-a5fb-f60718c906a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1"
Oct 30 19:45:05.197070 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.197047    4320 generic.go:356] "Generic (PLEG): container finished" podID="78f0bf84-4d74-451f-b984-36431daa554b" containerID="f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a" exitCode=60
Oct 30 19:45:05.197178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.197115    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerDied","Data":"f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"}
Oct 30 19:45:05.197335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.197316    4320 scope.go:122] "RemoveContainer" containerID="f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"
Oct 30 19:45:05.197567 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.197547    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8_odh-ai-gateway-infra(78f0bf84-4d74-451f-b984-36431daa554b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" podUID="78f0bf84-4d74-451f-b984-36431daa554b"
Oct 30 19:45:05.199082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.199064    4320 generic.go:356] "Generic (PLEG): container finished" podID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerID="302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68" exitCode=60
Oct 30 19:45:05.199194 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.199121    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerDied","Data":"302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68"}
Oct 30 19:45:05.199364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.199350    4320 scope.go:122] "RemoveContainer" containerID="302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68"
Oct 30 19:45:05.199630 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.199611    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph_odh-ai-gateway-infra(2416ca6b-67ff-4900-bb14-aa9c3912e909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909"
Oct 30 19:45:05.201096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.201072    4320 generic.go:356] "Generic (PLEG): container finished" podID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerID="a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1" exitCode=60
Oct 30 19:45:05.201166 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.201130    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerDied","Data":"a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"}
Oct 30 19:45:05.201395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.201378    4320 scope.go:122] "RemoveContainer" containerID="a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"
Oct 30 19:45:05.201764 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:05.201738    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q_odh-ai-gateway-infra(4ca30a1f-820f-48dc-9cb1-e8d02d365b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d"
Oct 30 19:45:05.210113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.210096    4320 scope.go:122] "RemoveContainer" containerID="884f7020dbd9a63a8bbf81894d84ce3026774beeda8f481bb67f8877364bc204"
Oct 30 19:45:05.224838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.224818    4320 scope.go:122] "RemoveContainer" containerID="5c84e65c36fca3140237eeb111ebabded136008b2dde84f1996ff86d08fd680c"
Oct 30 19:45:05.237973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.237957    4320 scope.go:122] "RemoveContainer" containerID="3b82ace67bc24204ef1434e3142a0ef0d6fa2c0b2cf6175c1d14465eeffa31ac"
Oct 30 19:45:05.251630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.251604    4320 scope.go:122] "RemoveContainer" containerID="2f7aa7a918d2e2830cae87f85bead552ce7357f0764a6b40c4dbfd9c5e79d53e"
Oct 30 19:45:05.265026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.264996    4320 scope.go:122] "RemoveContainer" containerID="4dda27fcbdff3d5436c7e326163dba72adf7116ffebf51fa182de4b7a25bb478"
Oct 30 19:45:05.279113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.278549    4320 scope.go:122] "RemoveContainer" containerID="b9725ffb234c13920c61fb10e9c5c4452d18a8f4c43f7d5058ad029df74a2fc5"
Oct 30 19:45:05.293265 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.293242    4320 scope.go:122] "RemoveContainer" containerID="9c432c03de74e1a23149e6abeca3ab2b2f78e2d8867c45a01e91987d9da88731"
Oct 30 19:45:05.308283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.308265    4320 scope.go:122] "RemoveContainer" containerID="666d7c1fb7cd55fb666ba3bd821dd31dbf9f60f44406adbb0dfc3c3919935716"
Oct 30 19:45:05.321931 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:05.321898    4320 scope.go:122] "RemoveContainer" containerID="d542f68be26b9f8a930bc343bd062d053239a6aeef1074a525669d3fe274c7df"
Oct 30 19:45:06.098176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.098140    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e654e480-05b3-4c1f-b984-f7997d43beee" path="/var/lib/kubelet/pods/e654e480-05b3-4c1f-b984-f7997d43beee/volumes"
Oct 30 19:45:06.211371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.211335    4320 scope.go:122] "RemoveContainer" containerID="cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"
Oct 30 19:45:06.211746 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.211714    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9_odh-ai-gateway-infra(2f4732d9-f4ab-4d7d-a5ad-49c97692c41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b"
Oct 30 19:45:06.213107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.213084    4320 scope.go:122] "RemoveContainer" containerID="851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"
Oct 30 19:45:06.213514 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.213420    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c_odh-ai-gateway-infra(ca784aee-fc45-4f3f-8300-06c9b686f604)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604"
Oct 30 19:45:06.217612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.214957    4320 scope.go:122] "RemoveContainer" containerID="f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"
Oct 30 19:45:06.217612 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.215284    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7_odh-ai-gateway-infra(b61b3e6d-4189-4984-a862-89fb78fdd12c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c"
Oct 30 19:45:06.217612 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.217125    4320 scope.go:122] "RemoveContainer" containerID="5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a"
Oct 30 19:45:06.217612 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.217348    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv_odh-ai-gateway-infra(8f267b1c-3898-4d33-a0a4-f64b5fb9e769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769"
Oct 30 19:45:06.219112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.219091    4320 scope.go:122] "RemoveContainer" containerID="f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1"
Oct 30 19:45:06.219398 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.219369    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m_odh-ai-gateway-infra(49943423-8173-4ac7-bb6d-c0324d84d6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd"
Oct 30 19:45:06.220801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.220775    4320 scope.go:122] "RemoveContainer" containerID="cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"
Oct 30 19:45:06.221039 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.221019    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr_odh-ai-gateway-infra(88d6df1a-d2ca-4856-9535-7405a748f19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b"
Oct 30 19:45:06.222866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.222847    4320 scope.go:122] "RemoveContainer" containerID="3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f"
Oct 30 19:45:06.223099 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.223075    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr_odh-ai-gateway-infra(fde68579-b9aa-43d6-a5fb-f60718c906a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1"
Oct 30 19:45:06.224753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.224736    4320 scope.go:122] "RemoveContainer" containerID="f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"
Oct 30 19:45:06.224942 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.224925    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8_odh-ai-gateway-infra(78f0bf84-4d74-451f-b984-36431daa554b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" podUID="78f0bf84-4d74-451f-b984-36431daa554b"
Oct 30 19:45:06.227010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.226990    4320 scope.go:122] "RemoveContainer" containerID="302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68"
Oct 30 19:45:06.227269 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.227238    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph_odh-ai-gateway-infra(2416ca6b-67ff-4900-bb14-aa9c3912e909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909"
Oct 30 19:45:06.228878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.228862    4320 scope.go:122] "RemoveContainer" containerID="a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"
Oct 30 19:45:06.229110 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.229089    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q_odh-ai-gateway-infra(4ca30a1f-820f-48dc-9cb1-e8d02d365b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d"
Oct 30 19:45:06.230839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:06.230821    4320 scope.go:122] "RemoveContainer" containerID="c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580"
Oct 30 19:45:06.231025 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:06.231008    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9_odh-ai-gateway-infra(df18a818-7dd3-4ddc-a2e1-300af1320e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a"
Oct 30 19:45:11.480887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.480855    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"]
Oct 30 19:45:11.485413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.485395    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.513591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.513564    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:11.514587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.514553    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.514784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.514719    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mls6\" (UniqueName: \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.514897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.514873    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.520932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.520908    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"]
Oct 30 19:45:11.616991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.616926    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.617468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.617430    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.617712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.617614    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mls6\" (UniqueName: \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.618532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.618507    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.620770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.620742    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.645060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.642916    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mls6\" (UniqueName: \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\") pod \"maas-api-revoke-keys-e2e-inf-b-469724-zp89t\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:11.800382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:11.800297    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:12.104384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:12.104355    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"]
Oct 30 19:45:12.105766 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:12.105728    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a6084a_1568_4447_9619_cd1f815354ae.slice/crio-e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89 WatchSource:0}: Error finding container e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89: Status 404 returned error can't find the container with id e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89
Oct 30 19:45:12.281821 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:12.281787    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t" event={"ID":"94a6084a-1568-4447-9619-cd1f815354ae","Type":"ContainerStarted","Data":"e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89"}
Oct 30 19:45:13.292028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:13.291931    4320 generic.go:356] "Generic (PLEG): container finished" podID="94a6084a-1568-4447-9619-cd1f815354ae" containerID="3df4dc6d2347a1f08106aea298a48514012fa4f956cc05caf910a3277793edaa" exitCode=0
Oct 30 19:45:13.292028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:13.291975    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t" event={"ID":"94a6084a-1568-4447-9619-cd1f815354ae","Type":"ContainerDied","Data":"3df4dc6d2347a1f08106aea298a48514012fa4f956cc05caf910a3277793edaa"}
Oct 30 19:45:13.514072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:13.514034    4320 scope.go:122] "RemoveContainer" containerID="39a48d44b0da6af702219290d6f440389a56861855365bebd751e3b83882675d"
Oct 30 19:45:13.524929 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:13.524908    4320 scope.go:122] "RemoveContainer" containerID="9ff69108019b2a30c26f1572878c3dbde591e5ee8f2ff495ce3dc2e51817fefc"
Oct 30 19:45:14.470831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.470804    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:14.558784 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.558703    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\" (UniqueName: \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\") pod \"94a6084a-1568-4447-9619-cd1f815354ae\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") "
Oct 30 19:45:14.558928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.558897    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\") pod \"94a6084a-1568-4447-9619-cd1f815354ae\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") "
Oct 30 19:45:14.559019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.559000    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\") pod \"94a6084a-1568-4447-9619-cd1f815354ae\" (UID: \"94a6084a-1568-4447-9619-cd1f815354ae\") "
Oct 30 19:45:14.559420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.559366    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca" pod "94a6084a-1568-4447-9619-cd1f815354ae" (UID: "94a6084a-1568-4447-9619-cd1f815354ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:14.559863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.559827    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:14.561490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.561460    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6" pod "94a6084a-1568-4447-9619-cd1f815354ae" (UID: "94a6084a-1568-4447-9619-cd1f815354ae"). InnerVolumeSpecName "kube-api-access-2mls6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:14.561982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.561945    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca" pod "94a6084a-1568-4447-9619-cd1f815354ae" (UID: "94a6084a-1568-4447-9619-cd1f815354ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:14.661060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.661035    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a6084a-1568-4447-9619-cd1f815354ae-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:14.661060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:14.661057    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mls6\" (UniqueName: \"kubernetes.io/projected/94a6084a-1568-4447-9619-cd1f815354ae-kube-api-access-2mls6\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:15.089335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:15.089297    4320 scope.go:122] "RemoveContainer" containerID="f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"
Oct 30 19:45:15.311541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:15.311514    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"
Oct 30 19:45:15.311746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:15.311515    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t" event={"ID":"94a6084a-1568-4447-9619-cd1f815354ae","Type":"ContainerDied","Data":"e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89"}
Oct 30 19:45:15.311746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:15.311664    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89"
Oct 30 19:45:16.089979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.089942    4320 scope.go:122] "RemoveContainer" containerID="5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a"
Oct 30 19:45:16.090347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.090101    4320 scope.go:122] "RemoveContainer" containerID="851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"
Oct 30 19:45:16.090414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.090352    4320 scope.go:122] "RemoveContainer" containerID="3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f"
Oct 30 19:45:16.323213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.323176    4320 generic.go:356] "Generic (PLEG): container finished" podID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerID="1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a" exitCode=60
Oct 30 19:45:16.323383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.323271    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerDied","Data":"1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a"}
Oct 30 19:45:16.323383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.323337    4320 scope.go:122] "RemoveContainer" containerID="f7883e2c8ba6a3324e13614fa712964e57e11533dbce5a6171cf3dad3237c1ff"
Oct 30 19:45:16.324079 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:16.324053    4320 scope.go:122] "RemoveContainer" containerID="1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a"
Oct 30 19:45:16.324542 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:16.324499    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7_odh-ai-gateway-infra(b61b3e6d-4189-4984-a862-89fb78fdd12c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c"
Oct 30 19:45:17.089496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.089407    4320 scope.go:122] "RemoveContainer" containerID="f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1"
Oct 30 19:45:17.089650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.089578    4320 scope.go:122] "RemoveContainer" containerID="a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"
Oct 30 19:45:17.089761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.089744    4320 scope.go:122] "RemoveContainer" containerID="c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580"
Oct 30 19:45:17.089874 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.089856    4320 scope.go:122] "RemoveContainer" containerID="f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"
Oct 30 19:45:17.334960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.334897    4320 generic.go:356] "Generic (PLEG): container finished" podID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerID="acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f" exitCode=60
Oct 30 19:45:17.335306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.334969    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerDied","Data":"acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f"}
Oct 30 19:45:17.335306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.335032    4320 scope.go:122] "RemoveContainer" containerID="851059bf4ab4a932a6872d3c592c569fddc2bf8c62fcb7a47fb6d2a19b098225"
Oct 30 19:45:17.335499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.335467    4320 scope.go:122] "RemoveContainer" containerID="acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f"
Oct 30 19:45:17.335869 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:17.335847    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c_odh-ai-gateway-infra(ca784aee-fc45-4f3f-8300-06c9b686f604)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604"
Oct 30 19:45:17.339475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.339447    4320 generic.go:356] "Generic (PLEG): container finished" podID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerID="28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f" exitCode=60
Oct 30 19:45:17.339749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.339723    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerDied","Data":"28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f"}
Oct 30 19:45:17.340142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.340126    4320 scope.go:122] "RemoveContainer" containerID="28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f"
Oct 30 19:45:17.340443 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:17.340423    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv_odh-ai-gateway-infra(8f267b1c-3898-4d33-a0a4-f64b5fb9e769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769"
Oct 30 19:45:17.342474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.342447    4320 generic.go:356] "Generic (PLEG): container finished" podID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerID="b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c" exitCode=60
Oct 30 19:45:17.342565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.342512    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" event={"ID":"fde68579-b9aa-43d6-a5fb-f60718c906a1","Type":"ContainerDied","Data":"b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c"}
Oct 30 19:45:17.342908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.342891    4320 scope.go:122] "RemoveContainer" containerID="b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c"
Oct 30 19:45:17.343307 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:17.343260    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr_odh-ai-gateway-infra(fde68579-b9aa-43d6-a5fb-f60718c906a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1"
Oct 30 19:45:17.349304 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.349285    4320 scope.go:122] "RemoveContainer" containerID="5c7f32b1174633477a556b4f87476dbb0d16f4a167a99e8b5b2f567072c2e95a"
Oct 30 19:45:17.362719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.362699    4320 scope.go:122] "RemoveContainer" containerID="3ac6a27b5039eb9e462dcc97afca04bd067aafc75f37a877660460f747f3618f"
Oct 30 19:45:17.374502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.374476    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:17.682735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.682715    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:17.813665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.813620    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\") pod \"b61b3e6d-4189-4984-a862-89fb78fdd12c\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") "
Oct 30 19:45:17.813797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.813691    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\" (UniqueName: \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\") pod \"b61b3e6d-4189-4984-a862-89fb78fdd12c\" (UID: \"b61b3e6d-4189-4984-a862-89fb78fdd12c\") "
Oct 30 19:45:17.813922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.813900    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca" pod "b61b3e6d-4189-4984-a862-89fb78fdd12c" (UID: "b61b3e6d-4189-4984-a862-89fb78fdd12c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.814230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.814213    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b3e6d-4189-4984-a862-89fb78fdd12c-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.815662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.815592    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4" pod "b61b3e6d-4189-4984-a862-89fb78fdd12c" (UID: "b61b3e6d-4189-4984-a862-89fb78fdd12c"). InnerVolumeSpecName "kube-api-access-62hz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.916231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:17.916200    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62hz4\" (UniqueName: \"kubernetes.io/projected/b61b3e6d-4189-4984-a862-89fb78fdd12c-kube-api-access-62hz4\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.352068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.352039    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerStarted","Data":"444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816"}
Oct 30 19:45:18.354117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.354094    4320 generic.go:356] "Generic (PLEG): container finished" podID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerID="f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06" exitCode=60
Oct 30 19:45:18.354241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.354169    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" event={"ID":"df18a818-7dd3-4ddc-a2e1-300af1320e2a","Type":"ContainerDied","Data":"f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06"}
Oct 30 19:45:18.354241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.354227    4320 scope.go:122] "RemoveContainer" containerID="c49cd711af37157cbc79d230ee514a6cefb47aca4004c5dbfba84501b4426580"
Oct 30 19:45:18.354611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.354576    4320 scope.go:122] "RemoveContainer" containerID="f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06"
Oct 30 19:45:18.354925 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:18.354900    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9_odh-ai-gateway-infra(df18a818-7dd3-4ddc-a2e1-300af1320e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a"
Oct 30 19:45:18.357571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.357552    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"
Oct 30 19:45:18.357571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.357560    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7" event={"ID":"b61b3e6d-4189-4984-a862-89fb78fdd12c","Type":"ContainerDied","Data":"2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea"}
Oct 30 19:45:18.362036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.362015    4320 generic.go:356] "Generic (PLEG): container finished" podID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerID="d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993" exitCode=60
Oct 30 19:45:18.362114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.362080    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" event={"ID":"49943423-8173-4ac7-bb6d-c0324d84d6dd","Type":"ContainerDied","Data":"d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993"}
Oct 30 19:45:18.362451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.362435    4320 scope.go:122] "RemoveContainer" containerID="d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993"
Oct 30 19:45:18.362767 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:18.362746    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m_odh-ai-gateway-infra(49943423-8173-4ac7-bb6d-c0324d84d6dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd"
Oct 30 19:45:18.365343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.365325    4320 generic.go:356] "Generic (PLEG): container finished" podID="78f0bf84-4d74-451f-b984-36431daa554b" containerID="63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c" exitCode=60
Oct 30 19:45:18.365424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.365350    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerDied","Data":"63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c"}
Oct 30 19:45:18.365645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.365630    4320 scope.go:122] "RemoveContainer" containerID="63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c"
Oct 30 19:45:18.365861 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:18.365845    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8_odh-ai-gateway-infra(78f0bf84-4d74-451f-b984-36431daa554b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" podUID="78f0bf84-4d74-451f-b984-36431daa554b"
Oct 30 19:45:18.390458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.390435    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:18.483848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.483800    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:18.533933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.533862    4320 scope.go:122] "RemoveContainer" containerID="1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a"
Oct 30 19:45:18.545567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.545539    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:18.565431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.565408    4320 scope.go:122] "RemoveContainer" containerID="f9ed065cda726f6dc52022355f925a952b73c05ea6b50e9d2b79eedb8d48d7f1"
Oct 30 19:45:18.612906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.612829    4320 scope.go:122] "RemoveContainer" containerID="f359653c0d0f3b7e24dbef1a16a61f82007fb447ffd2ad3a57eacff487f6975a"
Oct 30 19:45:18.614672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.614648    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:18.711071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.711038    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:18.712266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.712239    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:18.724004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.723976    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f44f74-29889825-s56p7"]
Oct 30 19:45:18.735268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.735246    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\") pod \"ca784aee-fc45-4f3f-8300-06c9b686f604\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") "
Oct 30 19:45:18.735382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.735364    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\" (UniqueName: \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\") pod \"ca784aee-fc45-4f3f-8300-06c9b686f604\" (UID: \"ca784aee-fc45-4f3f-8300-06c9b686f604\") "
Oct 30 19:45:18.735688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.735655    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca" pod "ca784aee-fc45-4f3f-8300-06c9b686f604" (UID: "ca784aee-fc45-4f3f-8300-06c9b686f604"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.736269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.736248    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca784aee-fc45-4f3f-8300-06c9b686f604-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.738408 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.738373    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59" pod "ca784aee-fc45-4f3f-8300-06c9b686f604" (UID: "ca784aee-fc45-4f3f-8300-06c9b686f604"). InnerVolumeSpecName "kube-api-access-xld59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.772013 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.771984    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:18.837208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.837138    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\" (UniqueName: \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\") pod \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") "
Oct 30 19:45:18.837356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.837241    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\") pod \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\" (UID: \"8f267b1c-3898-4d33-a0a4-f64b5fb9e769\") "
Oct 30 19:45:18.838463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.837994    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca" pod "8f267b1c-3898-4d33-a0a4-f64b5fb9e769" (UID: "8f267b1c-3898-4d33-a0a4-f64b5fb9e769"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.838463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.838103    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xld59\" (UniqueName: \"kubernetes.io/projected/ca784aee-fc45-4f3f-8300-06c9b686f604-kube-api-access-xld59\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.838463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.838118    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.840492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.840462    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b" pod "8f267b1c-3898-4d33-a0a4-f64b5fb9e769" (UID: "8f267b1c-3898-4d33-a0a4-f64b5fb9e769"). InnerVolumeSpecName "kube-api-access-x526b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.939233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.939205    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\" (UniqueName: \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\") pod \"fde68579-b9aa-43d6-a5fb-f60718c906a1\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") "
Oct 30 19:45:18.939455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.939440    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\") pod \"fde68579-b9aa-43d6-a5fb-f60718c906a1\" (UID: \"fde68579-b9aa-43d6-a5fb-f60718c906a1\") "
Oct 30 19:45:18.939825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.939799    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca" pod "fde68579-b9aa-43d6-a5fb-f60718c906a1" (UID: "fde68579-b9aa-43d6-a5fb-f60718c906a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.940008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.939995    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x526b\" (UniqueName: \"kubernetes.io/projected/8f267b1c-3898-4d33-a0a4-f64b5fb9e769-kube-api-access-x526b\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.940047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.940012    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde68579-b9aa-43d6-a5fb-f60718c906a1-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:18.941335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:18.941310    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv" pod "fde68579-b9aa-43d6-a5fb-f60718c906a1" (UID: "fde68579-b9aa-43d6-a5fb-f60718c906a1"). InnerVolumeSpecName "kube-api-access-nf6tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.042296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.042228    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6tv\" (UniqueName: \"kubernetes.io/projected/fde68579-b9aa-43d6-a5fb-f60718c906a1-kube-api-access-nf6tv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:19.091084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.090071    4320 scope.go:122] "RemoveContainer" containerID="302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68"
Oct 30 19:45:19.091084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.090268    4320 scope.go:122] "RemoveContainer" containerID="cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"
Oct 30 19:45:19.091084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.090403    4320 scope.go:122] "RemoveContainer" containerID="cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"
Oct 30 19:45:19.375898 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.375840    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"
Oct 30 19:45:19.375898 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.375860    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr" event={"ID":"fde68579-b9aa-43d6-a5fb-f60718c906a1","Type":"ContainerDied","Data":"338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23"}
Oct 30 19:45:19.376310 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.375930    4320 scope.go:122] "RemoveContainer" containerID="b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c"
Oct 30 19:45:19.379967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.379942    4320 generic.go:356] "Generic (PLEG): container finished" podID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerID="444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816" exitCode=60
Oct 30 19:45:19.380085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.380012    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerDied","Data":"444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816"}
Oct 30 19:45:19.380345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.380330    4320 scope.go:122] "RemoveContainer" containerID="444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816"
Oct 30 19:45:19.380647 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:19.380624    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q_odh-ai-gateway-infra(4ca30a1f-820f-48dc-9cb1-e8d02d365b5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d"
Oct 30 19:45:19.384447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.384429    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"
Oct 30 19:45:19.384556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.384450    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c" event={"ID":"ca784aee-fc45-4f3f-8300-06c9b686f604","Type":"ContainerDied","Data":"f00c866d31df8eb3a26a1379759bf67c8439d32de3a155aecbaa3f0ab2f989de"}
Oct 30 19:45:19.387117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.387092    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"
Oct 30 19:45:19.387212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.387133    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv" event={"ID":"8f267b1c-3898-4d33-a0a4-f64b5fb9e769","Type":"ContainerDied","Data":"f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4"}
Oct 30 19:45:19.390009 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.389994    4320 scope.go:122] "RemoveContainer" containerID="a545c04d4fdd185d6102b7b7693a9009cbfa68d72119bbe4c59601601500a7b1"
Oct 30 19:45:19.408977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.408958    4320 scope.go:122] "RemoveContainer" containerID="acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f"
Oct 30 19:45:19.415084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.415062    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:19.422503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.422481    4320 scope.go:122] "RemoveContainer" containerID="28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f"
Oct 30 19:45:19.464147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.464123    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:19.471299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.471278    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f6cef-29889825-2jtnr"]
Oct 30 19:45:19.497201 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.497178    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:19.545743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.545714    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:19.550501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.550476    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889825-nnqgv"]
Oct 30 19:45:19.559919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.559895    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:19.612963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.612934    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:19.618970 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.618939    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-287ea2-29889825-8sr2c"]
Oct 30 19:45:19.623689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.623638    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:19.698817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.698790    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:19.795855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.794768    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:19.858123 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.858091    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\" (UniqueName: \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\") pod \"49943423-8173-4ac7-bb6d-c0324d84d6dd\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") "
Oct 30 19:45:19.858317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.858297    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\") pod \"49943423-8173-4ac7-bb6d-c0324d84d6dd\" (UID: \"49943423-8173-4ac7-bb6d-c0324d84d6dd\") "
Oct 30 19:45:19.858677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.858648    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca" pod "49943423-8173-4ac7-bb6d-c0324d84d6dd" (UID: "49943423-8173-4ac7-bb6d-c0324d84d6dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.859833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.859811    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49943423-8173-4ac7-bb6d-c0324d84d6dd-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:19.861063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.861038    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5" pod "49943423-8173-4ac7-bb6d-c0324d84d6dd" (UID: "49943423-8173-4ac7-bb6d-c0324d84d6dd"). InnerVolumeSpecName "kube-api-access-njzf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.869558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.869533    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:19.961376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.961299    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\" (UniqueName: \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\") pod \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") "
Oct 30 19:45:19.961520 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.961500    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\") pod \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\" (UID: \"df18a818-7dd3-4ddc-a2e1-300af1320e2a\") "
Oct 30 19:45:19.961582 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.961545    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\") pod \"78f0bf84-4d74-451f-b984-36431daa554b\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") "
Oct 30 19:45:19.961677 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.961658    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\" (UniqueName: \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\") pod \"78f0bf84-4d74-451f-b984-36431daa554b\" (UID: \"78f0bf84-4d74-451f-b984-36431daa554b\") "
Oct 30 19:45:19.962129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.961932    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca" pod "df18a818-7dd3-4ddc-a2e1-300af1320e2a" (UID: "df18a818-7dd3-4ddc-a2e1-300af1320e2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.962129 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.962001    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca" pod "78f0bf84-4d74-451f-b984-36431daa554b" (UID: "78f0bf84-4d74-451f-b984-36431daa554b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.962422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.962399    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzf5\" (UniqueName: \"kubernetes.io/projected/49943423-8173-4ac7-bb6d-c0324d84d6dd-kube-api-access-njzf5\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:19.962485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.962428    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df18a818-7dd3-4ddc-a2e1-300af1320e2a-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:19.962485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.962446    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0bf84-4d74-451f-b984-36431daa554b-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:19.963675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.963649    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr" pod "df18a818-7dd3-4ddc-a2e1-300af1320e2a" (UID: "df18a818-7dd3-4ddc-a2e1-300af1320e2a"). InnerVolumeSpecName "kube-api-access-mtrhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.964059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:19.964037    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph" pod "78f0bf84-4d74-451f-b984-36431daa554b" (UID: "78f0bf84-4d74-451f-b984-36431daa554b"). InnerVolumeSpecName "kube-api-access-kn8ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.064809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.064774    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtrhr\" (UniqueName: \"kubernetes.io/projected/df18a818-7dd3-4ddc-a2e1-300af1320e2a-kube-api-access-mtrhr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:20.064809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.064809    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8ph\" (UniqueName: \"kubernetes.io/projected/78f0bf84-4d74-451f-b984-36431daa554b-kube-api-access-kn8ph\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:20.100754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.100720    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" path="/var/lib/kubelet/pods/8f267b1c-3898-4d33-a0a4-f64b5fb9e769/volumes"
Oct 30 19:45:20.101489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.101464    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" path="/var/lib/kubelet/pods/b61b3e6d-4189-4984-a862-89fb78fdd12c/volumes"
Oct 30 19:45:20.102131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.102103    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" path="/var/lib/kubelet/pods/ca784aee-fc45-4f3f-8300-06c9b686f604/volumes"
Oct 30 19:45:20.103189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.103166    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" path="/var/lib/kubelet/pods/fde68579-b9aa-43d6-a5fb-f60718c906a1/volumes"
Oct 30 19:45:20.398587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.398557    4320 generic.go:356] "Generic (PLEG): container finished" podID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerID="71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7" exitCode=60
Oct 30 19:45:20.399028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.398658    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerDied","Data":"71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7"}
Oct 30 19:45:20.399028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.398775    4320 scope.go:122] "RemoveContainer" containerID="cdf3373729485ea800f432de3d6191971f67c1836ae0e1efd470a599c7b2e542"
Oct 30 19:45:20.399189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.399175    4320 scope.go:122] "RemoveContainer" containerID="71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7"
Oct 30 19:45:20.399528 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:20.399501    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9_odh-ai-gateway-infra(2f4732d9-f4ab-4d7d-a5ad-49c97692c41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b"
Oct 30 19:45:20.402330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.402306    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m" event={"ID":"49943423-8173-4ac7-bb6d-c0324d84d6dd","Type":"ContainerDied","Data":"10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb"}
Oct 30 19:45:20.402431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.402317    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"
Oct 30 19:45:20.404528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.404507    4320 generic.go:356] "Generic (PLEG): container finished" podID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerID="68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189" exitCode=60
Oct 30 19:45:20.404645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.404572    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerDied","Data":"68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189"}
Oct 30 19:45:20.405017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.405001    4320 scope.go:122] "RemoveContainer" containerID="68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189"
Oct 30 19:45:20.405315 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:20.405292    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr_odh-ai-gateway-infra(88d6df1a-d2ca-4856-9535-7405a748f19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b"
Oct 30 19:45:20.407497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.407481    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"
Oct 30 19:45:20.407642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.407490    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8" event={"ID":"78f0bf84-4d74-451f-b984-36431daa554b","Type":"ContainerDied","Data":"fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00"}
Oct 30 19:45:20.409753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.409732    4320 generic.go:356] "Generic (PLEG): container finished" podID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerID="7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f" exitCode=60
Oct 30 19:45:20.409857 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.409802    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerDied","Data":"7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f"}
Oct 30 19:45:20.410165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.410150    4320 scope.go:122] "RemoveContainer" containerID="7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f"
Oct 30 19:45:20.410417 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:20.410398    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph_odh-ai-gateway-infra(2416ca6b-67ff-4900-bb14-aa9c3912e909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909"
Oct 30 19:45:20.412842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.412813    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9" event={"ID":"df18a818-7dd3-4ddc-a2e1-300af1320e2a","Type":"ContainerDied","Data":"97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4"}
Oct 30 19:45:20.412913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.412846    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"
Oct 30 19:45:20.413720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.413703    4320 scope.go:122] "RemoveContainer" containerID="d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993"
Oct 30 19:45:20.423691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.423668    4320 scope.go:122] "RemoveContainer" containerID="cf0c455cd48fb8f3d6a57aa6e2943abf5ab6a5132abe5874b60c22b60fbabd4a"
Oct 30 19:45:20.436918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.436899    4320 scope.go:122] "RemoveContainer" containerID="63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c"
Oct 30 19:45:20.448365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.448348    4320 scope.go:122] "RemoveContainer" containerID="302fc909dc14ab08498919779610543bf8abb6268543b78c0d1850b78caefa68"
Oct 30 19:45:20.462445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.462422    4320 scope.go:122] "RemoveContainer" containerID="f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06"
Oct 30 19:45:20.476956 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.476931    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:20.483211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.483187    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac614-29889825-hl86m"]
Oct 30 19:45:20.562722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.562693    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:20.574562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.574539    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3504b-29889825-z77q8"]
Oct 30 19:45:20.588477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.588456    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:20.653777 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.653750    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:20.672934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.672903    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\" (UniqueName: \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\") pod \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") "
Oct 30 19:45:20.673114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.673098    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\") pod \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\" (UID: \"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d\") "
Oct 30 19:45:20.673410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.673386    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08bcfe-29889825-ztns9"]
Oct 30 19:45:20.673507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.673465    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca" pod "4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" (UID: "4ca30a1f-820f-48dc-9cb1-e8d02d365b5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.673779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.673760    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:20.674813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.674784    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n" pod "4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" (UID: "4ca30a1f-820f-48dc-9cb1-e8d02d365b5d"). InnerVolumeSpecName "kube-api-access-r579n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.775742 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:20.775705    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r579n\" (UniqueName: \"kubernetes.io/projected/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d-kube-api-access-r579n\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:21.431881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.431797    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"
Oct 30 19:45:21.431881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.431808    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q" event={"ID":"4ca30a1f-820f-48dc-9cb1-e8d02d365b5d","Type":"ContainerDied","Data":"c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c"}
Oct 30 19:45:21.431881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.431857    4320 scope.go:122] "RemoveContainer" containerID="444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816"
Oct 30 19:45:21.477410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.477381    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:21.526983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.526950    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:21.557852 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.557809    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889825-rh27q"]
Oct 30 19:45:21.564132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.564108    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:21.628339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.628301    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:21.680858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.680824    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:21.793339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.793270    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\" (UniqueName: \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\") pod \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") "
Oct 30 19:45:21.793913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.793884    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\") pod \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\" (UID: \"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b\") "
Oct 30 19:45:21.794292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.794265    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca" pod "2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" (UID: "2f4732d9-f4ab-4d7d-a5ad-49c97692c41b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.794767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.794742    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:21.796547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.796278    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm" pod "2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" (UID: "2f4732d9-f4ab-4d7d-a5ad-49c97692c41b"). InnerVolumeSpecName "kube-api-access-ljgvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.836818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.836795    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:21.839211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.839190    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:21.896581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.896555    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgvm\" (UniqueName: \"kubernetes.io/projected/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b-kube-api-access-ljgvm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:21.998106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998018    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\") pod \"88d6df1a-d2ca-4856-9535-7405a748f19b\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") "
Oct 30 19:45:21.998268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998131    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\" (UniqueName: \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\") pod \"88d6df1a-d2ca-4856-9535-7405a748f19b\" (UID: \"88d6df1a-d2ca-4856-9535-7405a748f19b\") "
Oct 30 19:45:21.998268 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998214    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\" (UniqueName: \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\") pod \"2416ca6b-67ff-4900-bb14-aa9c3912e909\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") "
Oct 30 19:45:21.998401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998316    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\") pod \"2416ca6b-67ff-4900-bb14-aa9c3912e909\" (UID: \"2416ca6b-67ff-4900-bb14-aa9c3912e909\") "
Oct 30 19:45:21.998919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998879    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca" pod "88d6df1a-d2ca-4856-9535-7405a748f19b" (UID: "88d6df1a-d2ca-4856-9535-7405a748f19b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.999036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.998945    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca" pod "2416ca6b-67ff-4900-bb14-aa9c3912e909" (UID: "2416ca6b-67ff-4900-bb14-aa9c3912e909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.999152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.999129    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6df1a-d2ca-4856-9535-7405a748f19b-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:21.999152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:21.999152    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2416ca6b-67ff-4900-bb14-aa9c3912e909-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.000921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.000898    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz" pod "88d6df1a-d2ca-4856-9535-7405a748f19b" (UID: "88d6df1a-d2ca-4856-9535-7405a748f19b"). InnerVolumeSpecName "kube-api-access-rr5nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.001205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.001187    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g" pod "2416ca6b-67ff-4900-bb14-aa9c3912e909" (UID: "2416ca6b-67ff-4900-bb14-aa9c3912e909"). InnerVolumeSpecName "kube-api-access-qz85g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.099334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.099298    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" path="/var/lib/kubelet/pods/49943423-8173-4ac7-bb6d-c0324d84d6dd/volumes"
Oct 30 19:45:22.099819 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.099796    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" path="/var/lib/kubelet/pods/4ca30a1f-820f-48dc-9cb1-e8d02d365b5d/volumes"
Oct 30 19:45:22.100250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.100230    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f0bf84-4d74-451f-b984-36431daa554b" path="/var/lib/kubelet/pods/78f0bf84-4d74-451f-b984-36431daa554b/volumes"
Oct 30 19:45:22.100342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.100328    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5nz\" (UniqueName: \"kubernetes.io/projected/88d6df1a-d2ca-4856-9535-7405a748f19b-kube-api-access-rr5nz\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.100402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.100346    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz85g\" (UniqueName: \"kubernetes.io/projected/2416ca6b-67ff-4900-bb14-aa9c3912e909-kube-api-access-qz85g\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.101107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.101085    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" path="/var/lib/kubelet/pods/df18a818-7dd3-4ddc-a2e1-300af1320e2a/volumes"
Oct 30 19:45:22.446249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.446208    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr" event={"ID":"88d6df1a-d2ca-4856-9535-7405a748f19b","Type":"ContainerDied","Data":"9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b"}
Oct 30 19:45:22.447000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.446308    4320 scope.go:122] "RemoveContainer" containerID="68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189"
Oct 30 19:45:22.447000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.446228    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"
Oct 30 19:45:22.449396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.449373    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"
Oct 30 19:45:22.449522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.449393    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph" event={"ID":"2416ca6b-67ff-4900-bb14-aa9c3912e909","Type":"ContainerDied","Data":"85b6be531bb9703c4e176b0b67b168985c268065e5bb9f67ab4a8db0f1e68a35"}
Oct 30 19:45:22.452455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.452433    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"
Oct 30 19:45:22.452642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.452450    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9" event={"ID":"2f4732d9-f4ab-4d7d-a5ad-49c97692c41b","Type":"ContainerDied","Data":"1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993"}
Oct 30 19:45:22.458696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.458680    4320 scope.go:122] "RemoveContainer" containerID="7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f"
Oct 30 19:45:22.468375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.468360    4320 scope.go:122] "RemoveContainer" containerID="71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7"
Oct 30 19:45:22.501176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.501150    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:22.512570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.512548    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-249dd4-29889825-xfqkr"]
Oct 30 19:45:22.537286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.537262    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:22.541510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.541485    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-469724-29889825-mfsph"]
Oct 30 19:45:22.573328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.573309    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:22.584003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:22.583984    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889825-vxlw9"]
Oct 30 19:45:23.033484 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:23.033444    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"]
Oct 30 19:45:23.038547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:23.038519    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9e55ba-7pszd"]
Oct 30 19:45:24.097820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:24.097787    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" path="/var/lib/kubelet/pods/2416ca6b-67ff-4900-bb14-aa9c3912e909/volumes"
Oct 30 19:45:24.098247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:24.098232    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" path="/var/lib/kubelet/pods/2f4732d9-f4ab-4d7d-a5ad-49c97692c41b/volumes"
Oct 30 19:45:24.098696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:24.098682    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7158e90c-3e60-4f65-a1b1-ae115af59943" path="/var/lib/kubelet/pods/7158e90c-3e60-4f65-a1b1-ae115af59943/volumes"
Oct 30 19:45:24.099365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:24.099350    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" path="/var/lib/kubelet/pods/88d6df1a-d2ca-4856-9535-7405a748f19b/volumes"
Oct 30 19:45:25.231122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:25.231074    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"]
Oct 30 19:45:25.242785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:25.242752    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-469724-zp89t"]
Oct 30 19:45:26.098163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:26.098128    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a6084a-1568-4447-9619-cd1f815354ae" path="/var/lib/kubelet/pods/94a6084a-1568-4447-9619-cd1f815354ae/volumes"
Oct 30 19:45:28.567480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:28.567439    4320 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-217.ec2.internal" podUID="32feda196b1df490b5d951791a17beab" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:28.644509 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:28.644469    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:45:32.534365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:32.534328    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:45:32.539097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:32.539063    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy" containerID="cri-o://c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771" gracePeriod=30
Oct 30 19:45:34.206166 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206126    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-conmon-444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-conmon-444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206244    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-conmon-63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-conmon-63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206266    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-conmon-f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-conmon-f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206385    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b3e6d_4189_4984_a862_89fb78fdd12c.slice/crio-1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a.scope WatchSource:0}: Error finding container 1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a: Status 404 returned error can't find the container with id 1f4139f83153c0fcf5a72212990e711e15a908bdbf2bdb044130088f1ba5f00a
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206416    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-444948a0e6ffdad6d25bb60767d3247da0d45ff2ae3faf675715c3a15d7ce816.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206441    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-63e42beed99ab33c028b0604dffed38c4c1915dc4a71360f68ea631a7ae3ec2c.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206474    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-f6acea5fb967ad3b97f6e5dca387ebba8a7bb53d8ba1e90e6d4647dfd7268a06.scope: no such file or directory
Oct 30 19:45:34.206629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.206508    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49943423_8173_4ac7_bb6d_c0324d84d6dd.slice/crio-d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49943423_8173_4ac7_bb6d_c0324d84d6dd.slice/crio-d174597c1f5a07d59a7a67d1c3c28f2831393607d507466e72868f3a6d217993.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207206    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice/crio-conmon-7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice/crio-conmon-7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207242    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-conmon-68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-conmon-68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207264    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-conmon-71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-conmon-71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207283    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-71cf7f4cbaa192ca132dfd3dbc665c48ae17c9549419f2a66902b675e59078f7.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207304    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice/crio-7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice/crio-7c074ec5f16c29e72f01f080eeb47de6c0b51fe5933be5b22d3e697f7c96007f.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207321    4320 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-68491a7aad01c825f9ef32f8aec09a29137744a7874bd09e1c16e086c6d87189.scope: no such file or directory
Oct 30 19:45:34.207629 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207551    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f267b1c_3898_4d33_a0a4_f64b5fb9e769.slice/crio-28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f.scope WatchSource:0}: Error finding container 28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f: Status 404 returned error can't find the container with id 28c69e719372e934f094638070b984bd741f2b0ba6312013735101d952e9f07f
Oct 30 19:45:34.208082 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.207979    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca784aee_fc45_4f3f_8300_06c9b686f604.slice/crio-acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f.scope WatchSource:0}: Error finding container acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f: Status 404 returned error can't find the container with id acd64cd86baebb0a52e18552527565a425fad23acf0d0d230314c53fc1d3469f
Oct 30 19:45:34.208155 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:34.208133    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde68579_b9aa_43d6_a5fb_f60718c906a1.slice/crio-b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c.scope WatchSource:0}: Error finding container b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c: Status 404 returned error can't find the container with id b7a2b64dc0353124f9522cd4a346b5cfafc78c8b12ec03f59adf6f299dba6b2c
Oct 30 19:45:34.332134 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:34.332014    4320 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9886b8b7_4bd0_487b_af82_94b47fc2fd39.slice/crio-720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde68579_b9aa_43d6_a5fb_f60718c906a1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b3e6d_4189_4984_a862_89fb78fdd12c.slice/crio-2f7b86597ca9bff2f896b32092cc8cb439e562b1829f590172f72b0f388c18ea\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a6084a_1568_4447_9619_cd1f815354ae.slice/crio-conmon-3df4dc6d2347a1f08106aea298a48514012fa4f956cc05caf910a3277793edaa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a6084a_1568_4447_9619_cd1f815354ae.slice/crio-e00ace8a3bd5c0791fcb03d2620380489e31048d288f2aebd09770b94ad45b89\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49943423_8173_4ac7_bb6d_c0324d84d6dd.slice/crio-10af061a8e75ded85284f14b9c20248134c2003e75787e63fd55fb1c54b4ffcb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice/crio-97c302ce1a25358267ec324dbbcd432455b39a838b44f5e7cf9236f914ab3fd4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9886b8b7_4bd0_487b_af82_94b47fc2fd39.slice/crio-conmon-720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice/crio-1a4b573f912386179aa676e1f3481b5e9eabed6c549def5016d41ef696e62993\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice/crio-85b6be531bb9703c4e176b0b67b168985c268065e5bb9f67ab4a8db0f1e68a35\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde68579_b9aa_43d6_a5fb_f60718c906a1.slice/crio-338eba80994dca4f8a4e537a9f99ee85afa5e699e4a0f3a1dc7c5433df658c23\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice/crio-c28742418a4429aa728db59ba352e2a4cfe812c3e4bf724fcc6f0d28002a745c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49943423_8173_4ac7_bb6d_c0324d84d6dd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0bf84_4d74_451f_b984_36431daa554b.slice/crio-fc2cd39f93175706c8afdf7067fd633c911ddd643783157086dde35988404f00\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca784aee_fc45_4f3f_8300_06c9b686f604.slice/crio-f00c866d31df8eb3a26a1379759bf67c8439d32de3a155aecbaa3f0ab2f989de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice/crio-9d2df980404e7bbe98577db9c7a58d73411edf1bc3d245ef8d011df5dd47887b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2416ca6b_67ff_4900_bb14_aa9c3912e909.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18a818_7dd3_4ddc_a2e1_300af1320e2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b3e6d_4189_4984_a862_89fb78fdd12c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca784aee_fc45_4f3f_8300_06c9b686f604.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4732d9_f4ab_4d7d_a5ad_49c97692c41b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f267b1c_3898_4d33_a0a4_f64b5fb9e769.slice/crio-f0368484e6d9651d814326e5d7d8b2757c03f66e56c17d66c817b7881953fdb4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca30a1f_820f_48dc_9cb1_e8d02d365b5d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a6084a_1568_4447_9619_cd1f815354ae.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6df1a_d2ca_4856_9535_7405a748f19b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:34.480576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.480546    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:34.522813 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.522785    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\" (UniqueName: \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\") pod \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") "
Oct 30 19:45:34.522950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.522920    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\") pod \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\" (UID: \"9886b8b7-4bd0-487b-af82-94b47fc2fd39\") "
Oct 30 19:45:34.523265 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.523238    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca" pod "9886b8b7-4bd0-487b-af82-94b47fc2fd39" (UID: "9886b8b7-4bd0-487b-af82-94b47fc2fd39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:34.524705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.524685    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn" pod "9886b8b7-4bd0-487b-af82-94b47fc2fd39" (UID: "9886b8b7-4bd0-487b-af82-94b47fc2fd39"). InnerVolumeSpecName "kube-api-access-wj5dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:34.603962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.603933    4320 generic.go:356] "Generic (PLEG): container finished" podID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerID="720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20" exitCode=137
Oct 30 19:45:34.604183 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.604103    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" event={"ID":"9886b8b7-4bd0-487b-af82-94b47fc2fd39","Type":"ContainerDied","Data":"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"}
Oct 30 19:45:34.604183 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.604139    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr" event={"ID":"9886b8b7-4bd0-487b-af82-94b47fc2fd39","Type":"ContainerDied","Data":"64ce60e2962b61a006ed4add1c3f2c7b47066501582c0fdd0b25ffd3728ed122"}
Oct 30 19:45:34.604183 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.604159    4320 scope.go:122] "RemoveContainer" containerID="720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"
Oct 30 19:45:34.604346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.604274    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"
Oct 30 19:45:34.637622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.625412    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9886b8b7-4bd0-487b-af82-94b47fc2fd39-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:34.637622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.625442    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5dn\" (UniqueName: \"kubernetes.io/projected/9886b8b7-4bd0-487b-af82-94b47fc2fd39-kube-api-access-wj5dn\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:34.637622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.629393    4320 scope.go:122] "RemoveContainer" containerID="6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"
Oct 30 19:45:34.640473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.640453    4320 scope.go:122] "RemoveContainer" containerID="720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"
Oct 30 19:45:34.640838 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:34.640818    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20\": container with ID starting with 720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20 not found: ID does not exist" containerID="720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"
Oct 30 19:45:34.640910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.640847    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20"} err="failed to get container status \"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20\": rpc error: code = NotFound desc = could not find container \"720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20\": container with ID starting with 720e320effdc0e83357f689a31507582223ebd767e79367445bec6c22e20cc20 not found: ID does not exist"
Oct 30 19:45:34.640910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.640869    4320 scope.go:122] "RemoveContainer" containerID="6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"
Oct 30 19:45:34.641164 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:34.641146    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615\": container with ID starting with 6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615 not found: ID does not exist" containerID="6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"
Oct 30 19:45:34.641216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.641169    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615"} err="failed to get container status \"6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615\": rpc error: code = NotFound desc = could not find container \"6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615\": container with ID starting with 6c15bb53968b2363de48657d7af871a023304549f521efda1ad7514f4caea615 not found: ID does not exist"
Oct 30 19:45:34.661920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.661895    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:34.666732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:34.666710    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5d92fa-29889825-wcwkr"]
Oct 30 19:45:36.098626 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:36.098577    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" path="/var/lib/kubelet/pods/9886b8b7-4bd0-487b-af82-94b47fc2fd39/volumes"
Oct 30 19:45:37.891369 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.891341    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:45:37.974846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.974807    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.974858    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.974911    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.974964    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975072    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975135    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975130    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:37.975220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975202    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975267    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975435 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975337    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\") pod \"7465b832-8cb3-4d74-9604-2e66a10a5738\" (UID: \"7465b832-8cb3-4d74-9604-2e66a10a5738\") "
Oct 30 19:45:37.975621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975578    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:37.975801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975754    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:37.976212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.975925    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:37.976212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.976110    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:37.976212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.976180    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:37.976526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.976470    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7465b832-8cb3-4d74-9604-2e66a10a5738-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:37.976526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.976495    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:37.979385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.978738    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:37.979491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.979330    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "kube-api-access-lw6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:37.979491 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.979351    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:37.979869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:37.979844    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo" pod "7465b832-8cb3-4d74-9604-2e66a10a5738" (UID: "7465b832-8cb3-4d74-9604-2e66a10a5738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:38.077982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.077953    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6r2\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-kube-api-access-lw6r2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.077982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.077980    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.078125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.077989    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.078125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.077997    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.078125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.078005    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7465b832-8cb3-4d74-9604-2e66a10a5738-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.078125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.078013    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7465b832-8cb3-4d74-9604-2e66a10a5738-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:45:38.641861 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.641826    4320 generic.go:356] "Generic (PLEG): container finished" podID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerID="c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771" exitCode=0
Oct 30 19:45:38.642031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.641873    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"
Oct 30 19:45:38.642031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.641912    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerDied","Data":"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"}
Oct 30 19:45:38.642031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.641977    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf" event={"ID":"7465b832-8cb3-4d74-9604-2e66a10a5738","Type":"ContainerDied","Data":"bee26b8bcd458dc56ab2a8e5dd1a0f328066a75077ceda637e3fd62447fc5aed"}
Oct 30 19:45:38.642031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.642005    4320 scope.go:122] "RemoveContainer" containerID="c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"
Oct 30 19:45:38.653377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.653351    4320 scope.go:122] "RemoveContainer" containerID="3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"
Oct 30 19:45:38.663081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.663063    4320 scope.go:122] "RemoveContainer" containerID="c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"
Oct 30 19:45:38.663357 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:38.663333    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771\": container with ID starting with c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771 not found: ID does not exist" containerID="c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"
Oct 30 19:45:38.663453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.663363    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771"} err="failed to get container status \"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771\": rpc error: code = NotFound desc = could not find container \"c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771\": container with ID starting with c19ef831b59899df46d182823d78d19ab4d6e499cc5ce31e91386ec90b4bc771 not found: ID does not exist"
Oct 30 19:45:38.663453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.663380    4320 scope.go:122] "RemoveContainer" containerID="3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"
Oct 30 19:45:38.663662 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:38.663639    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4\": container with ID starting with 3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4 not found: ID does not exist" containerID="3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"
Oct 30 19:45:38.663715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.663667    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4"} err="failed to get container status \"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4\": rpc error: code = NotFound desc = could not find container \"3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4\": container with ID starting with 3adc466a085badf30c1d7f2cf4212f72777bf6d531e9eefa9108e70587bc8db4 not found: ID does not exist"
Oct 30 19:45:38.709893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.709868    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:45:38.721534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:38.721511    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-469724-openshift-default-6c499b47c7-jm8kf"]
Oct 30 19:45:40.099223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:40.099176    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" path="/var/lib/kubelet/pods/7465b832-8cb3-4d74-9604-2e66a10a5738/volumes"
Oct 30 19:45:50.033560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:50.033523    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"]
Oct 30 19:45:50.044555 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:50.044527    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d78251b-ncrz2"]
Oct 30 19:45:50.097695 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:50.097669    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7900f05-1b45-4a47-9072-c19a5b3e52b6" path="/var/lib/kubelet/pods/e7900f05-1b45-4a47-9072-c19a5b3e52b6/volumes"
Oct 30 19:45:56.314561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.314524    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"]
Oct 30 19:45:56.315269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315250    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315271    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315285    4320 cpu_manager.go:414] "containerMap: removing container" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315290    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315303    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315308    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.315314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315315    4320 cpu_manager.go:414] "containerMap: removing container" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315320    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315330    4320 cpu_manager.go:414] "containerMap: removing container" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315335    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315342    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315346    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315355    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315360    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315364    4320 cpu_manager.go:414] "containerMap: removing container" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315369    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315378    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315382    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315389    4320 cpu_manager.go:414] "containerMap: removing container" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315393    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315400    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315404    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315415    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315419    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315433    4320 cpu_manager.go:414] "containerMap: removing container" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315437    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315442    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315447    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315455    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315459    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315467    4320 cpu_manager.go:414] "containerMap: removing container" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315471    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315477    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315481    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315488    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315493    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315500    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.315497 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315505    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315513    4320 cpu_manager.go:414] "containerMap: removing container" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315517    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315525    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315529    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315535    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315539    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315546    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315550    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315561    4320 cpu_manager.go:414] "containerMap: removing container" podUID="94a6084a-1568-4447-9619-cd1f815354ae" containerName="revoke-keys"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315565    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a6084a-1568-4447-9619-cd1f815354ae" containerName="revoke-keys"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315571    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315575    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315581    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315586    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315609    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315617    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315623    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315628    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315635    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315639    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315739    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315749    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315754    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315758    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315764    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315769    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315774    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315779    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315784    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315789    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315794    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315799    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315804    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315812    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315817    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315821    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315828    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315832    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315839    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315843    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315848    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315854    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315859    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61b3e6d-4189-4984-a862-89fb78fdd12c" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315865    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315872    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315877    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a6084a-1568-4447-9619-cd1f815354ae" containerName="revoke-keys"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315883    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde68579-b9aa-43d6-a5fb-f60718c906a1" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315891    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:45:56.316426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.315896    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:45:56.320416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.320398    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.335317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.335288    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:56.339721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.339695    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"]
Oct 30 19:45:56.345950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.345928    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:45:56.390239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.390213    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.390379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.390245    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4kk\" (UniqueName: \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.390472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.390451    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.491924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.491891    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.492092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.492074    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.492165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.492108    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4kk\" (UniqueName: \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.492646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.492628    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.494393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.494370    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.510126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.510098    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4kk\" (UniqueName: \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\") pod \"maas-api-revoke-sub-c63ba737-7qzjc\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.636969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.636936    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:45:56.937920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:56.937895    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"]
Oct 30 19:45:56.938891 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:45:56.938855    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfa3f0e_74e5_445a_9a4e_d9eed949a5aa.slice/crio-581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007 WatchSource:0}: Error finding container 581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007: Status 404 returned error can't find the container with id 581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007
Oct 30 19:45:57.794536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:57.794508    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc" event={"ID":"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa","Type":"ContainerStarted","Data":"02b67584ada7d08dee14ae3a0386a0f7323a432b8a16d0cf493287725bcc5a2b"}
Oct 30 19:45:57.794967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:57.794556    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc" event={"ID":"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa","Type":"ContainerStarted","Data":"581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007"}
Oct 30 19:45:57.835513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:57.835468    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc" podStartSLOduration=1.8354556149999999 podStartE2EDuration="1.835455615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:57.832747099 +0000 UTC m=+2505.868259563" watchObservedRunningTime="2026-10-30 19:45:57.835455615 +0000 UTC m=+2505.870968078"
Oct 30 19:45:58.646070 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:45:58.646026    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:45:58.804149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:58.804114    4320 generic.go:356] "Generic (PLEG): container finished" podID="fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" containerID="02b67584ada7d08dee14ae3a0386a0f7323a432b8a16d0cf493287725bcc5a2b" exitCode=0
Oct 30 19:45:58.804511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:45:58.804194    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc" event={"ID":"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa","Type":"ContainerDied","Data":"02b67584ada7d08dee14ae3a0386a0f7323a432b8a16d0cf493287725bcc5a2b"}
Oct 30 19:46:00.011977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.011952    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:46:00.158967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.158939    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\" (UniqueName: \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\") pod \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") "
Oct 30 19:46:00.159115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.159032    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\") pod \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") "
Oct 30 19:46:00.159115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.159093    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\") pod \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\" (UID: \"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa\") "
Oct 30 19:46:00.159373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.159347    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca" pod "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" (UID: "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:00.159711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.159693    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:00.161004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.160985    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk" pod "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" (UID: "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa"). InnerVolumeSpecName "kube-api-access-9f4kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:00.161115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.161094    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca" pod "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" (UID: "fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:00.261061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.261037    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f4kk\" (UniqueName: \"kubernetes.io/projected/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-kube-api-access-9f4kk\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:00.261061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.261059    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:00.827935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.827907    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"
Oct 30 19:46:00.828099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.827946    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc" event={"ID":"fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa","Type":"ContainerDied","Data":"581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007"}
Oct 30 19:46:00.828099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:00.827992    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581768fad5eb469f17aac8e9daf925da52dc224dc0f19cbf0c744617106e7007"
Oct 30 19:46:06.686537 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.684234    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"]
Oct 30 19:46:06.688074 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688046    4320 cpu_manager.go:414] "containerMap: removing container" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:46:06.688219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688206    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:46:06.688301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688292    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" containerName="revoke-keys"
Oct 30 19:46:06.688376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688367    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" containerName="revoke-keys"
Oct 30 19:46:06.688466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688458    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:46:06.688535 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688526    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:46:06.688649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688639    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:46:06.688731 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688722    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:46:06.688820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688811    4320 cpu_manager.go:414] "containerMap: removing container" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:46:06.688896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688885    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:46:06.688967 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.688959    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:46:06.689037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689028    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:46:06.689114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689105    4320 cpu_manager.go:414] "containerMap: removing container" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:46:06.689183 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689174    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:46:06.689261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689252    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:46:06.689328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689319    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7465b832-8cb3-4d74-9604-2e66a10a5738" containerName="istio-proxy"
Oct 30 19:46:06.689399 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689391    4320 cpu_manager.go:414] "containerMap: removing container" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:46:06.689462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689453    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:46:06.689544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689535    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:46:06.689690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689677    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4732d9-f4ab-4d7d-a5ad-49c97692c41b" containerName="cleanup"
Oct 30 19:46:06.689976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.689964    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca30a1f-820f-48dc-9cb1-e8d02d365b5d" containerName="cleanup"
Oct 30 19:46:06.690065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690056    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="df18a818-7dd3-4ddc-a2e1-300af1320e2a" containerName="cleanup"
Oct 30 19:46:06.690144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690135    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="49943423-8173-4ac7-bb6d-c0324d84d6dd" containerName="cleanup"
Oct 30 19:46:06.690417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690396    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6df1a-d2ca-4856-9535-7405a748f19b" containerName="cleanup"
Oct 30 19:46:06.690417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690420    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2416ca6b-67ff-4900-bb14-aa9c3912e909" containerName="cleanup"
Oct 30 19:46:06.690417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690432    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca784aee-fc45-4f3f-8300-06c9b686f604" containerName="cleanup"
Oct 30 19:46:06.690734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690445    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f267b1c-3898-4d33-a0a4-f64b5fb9e769" containerName="cleanup"
Oct 30 19:46:06.690734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690453    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9886b8b7-4bd0-487b-af82-94b47fc2fd39" containerName="cleanup"
Oct 30 19:46:06.690734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690465    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f0bf84-4d74-451f-b984-36431daa554b" containerName="cleanup"
Oct 30 19:46:06.690734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.690473    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" containerName="revoke-keys"
Oct 30 19:46:06.697722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.697696    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.706298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.706271    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"]
Oct 30 19:46:06.726381 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.726352    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:06.726531 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.726394    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:06.764109 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.764081    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.764255 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.764152    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qpk\" (UniqueName: \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.764378 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.764358    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.865862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.865831    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.865979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.865935    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.866058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.866011    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qpk\" (UniqueName: \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.866572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.866549    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.868203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.868175    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:06.885413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:06.885387    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qpk\" (UniqueName: \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\") pod \"maas-api-revoke-sub-677830fa-67trg\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:07.024104 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.024029    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:07.026417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.026391    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:46:07.030674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.030654    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.049238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.049214    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-576a62\"/\"openshift-service-ca.crt\""
Oct 30 19:46:07.049556 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.049536    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-576a62\"/\"test-model-576a62-kserve-self-signed-certs\""
Oct 30 19:46:07.049796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.049778    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-576a62\"/\"kube-root-ca.crt\""
Oct 30 19:46:07.052660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.049855    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-576a62\"/\"default-dockercfg-p26xd\""
Oct 30 19:46:07.055723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.055701    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:46:07.068693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.068663    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lmq\" (UniqueName: \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.068822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.068797    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.068938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.068875    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.069029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.068990    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.069127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.069106    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.069209 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.069188    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171078    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171140    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171235    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171277    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171431    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lmq\" (UniqueName: \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171513    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.171614 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.171517    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.172464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.172441    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.172962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.172938    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.174823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.174801    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.176615 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.176581    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.191510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.191483    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lmq\" (UniqueName: \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\") pod \"test-model-576a62-kserve-56565fd85c-9vlcm\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") " pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.410955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.409458    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"]
Oct 30 19:46:07.410955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.409962    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:07.754017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.753987    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:46:07.754826 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:07.754799    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9f3e86b_6f24_4e7f_8bb9_88a4f36a32dc.slice/crio-3c00bef936c8792f01eff4c5a2ba08c61349344b9d0c378b1f7b0a07aeb8a484 WatchSource:0}: Error finding container 3c00bef936c8792f01eff4c5a2ba08c61349344b9d0c378b1f7b0a07aeb8a484: Status 404 returned error can't find the container with id 3c00bef936c8792f01eff4c5a2ba08c61349344b9d0c378b1f7b0a07aeb8a484
Oct 30 19:46:07.883947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.883915    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" event={"ID":"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc","Type":"ContainerStarted","Data":"3c00bef936c8792f01eff4c5a2ba08c61349344b9d0c378b1f7b0a07aeb8a484"}
Oct 30 19:46:07.886195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:07.886170    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg" event={"ID":"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d","Type":"ContainerStarted","Data":"5a2efaf8b319df50cc3c96b43b3adc43ad7f0d33f8da50a4075cfeaa50f830c5"}
Oct 30 19:46:08.897992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:08.897956    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" event={"ID":"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc","Type":"ContainerStarted","Data":"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"}
Oct 30 19:46:08.900208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:08.900178    4320 generic.go:356] "Generic (PLEG): container finished" podID="2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" containerID="5565cfaabf3b6463552e80aab406ee4c013084cfab026f95c06f2aa5b14eea15" exitCode=0
Oct 30 19:46:08.900342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:08.900260    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg" event={"ID":"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d","Type":"ContainerDied","Data":"5565cfaabf3b6463552e80aab406ee4c013084cfab026f95c06f2aa5b14eea15"}
Oct 30 19:46:08.937929 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:08.937877    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" podStartSLOduration=2.937862657 podStartE2EDuration="2.937862657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:08.936524581 +0000 UTC m=+2516.972037046" watchObservedRunningTime="2026-10-30 19:46:08.937862657 +0000 UTC m=+2516.973375121"
Oct 30 19:46:09.410414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:09.410383    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:10.136243 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.136211    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:10.249179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.249109    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\" (UniqueName: \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\") pod \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") "
Oct 30 19:46:10.249390 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.249369    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\") pod \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") "
Oct 30 19:46:10.249448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.249430    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\") pod \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\" (UID: \"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d\") "
Oct 30 19:46:10.249798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.249768    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca" pod "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" (UID: "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:10.250103 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.250076    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.251395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.251372    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk" pod "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" (UID: "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d"). InnerVolumeSpecName "kube-api-access-67qpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:10.251522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.251507    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca" pod "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" (UID: "2c36c531-792b-4aa3-b8a2-e08ba5d67a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:10.351779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.351756    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.351779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.351780    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qpk\" (UniqueName: \"kubernetes.io/projected/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d-kube-api-access-67qpk\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.921088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.921058    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"
Oct 30 19:46:10.921260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.921082    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg" event={"ID":"2c36c531-792b-4aa3-b8a2-e08ba5d67a6d","Type":"ContainerDied","Data":"5a2efaf8b319df50cc3c96b43b3adc43ad7f0d33f8da50a4075cfeaa50f830c5"}
Oct 30 19:46:10.921260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:10.921149    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2efaf8b319df50cc3c96b43b3adc43ad7f0d33f8da50a4075cfeaa50f830c5"
Oct 30 19:46:11.393106 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.393076    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"]
Oct 30 19:46:11.393762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.393744    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" containerName="revoke-keys"
Oct 30 19:46:11.393762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.393760    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" containerName="revoke-keys"
Oct 30 19:46:11.393901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.393861    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" containerName="revoke-keys"
Oct 30 19:46:11.397258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.397241    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.415221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.415195    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"]
Oct 30 19:46:11.421841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.421819    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:11.421968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.421848    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:11.465754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.465726    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.465902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.465869    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.466024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.466009    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vch6g\" (UniqueName: \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.567297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.567266    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vch6g\" (UniqueName: \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.567437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.567365    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.567437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.567423    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.568031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.568012    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.570569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.570543    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.595734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.595711    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vch6g\" (UniqueName: \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\") pod \"maas-api-revoke-sub-f6fa8a8b-h7xv8\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:11.714023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:11.713961    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:12.020355 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.020331    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"]
Oct 30 19:46:12.021232 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:12.021204    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492bf6de_78cb_4039_b875_a8b337cb255f.slice/crio-0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7 WatchSource:0}: Error finding container 0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7: Status 404 returned error can't find the container with id 0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7
Oct 30 19:46:12.031670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.031644    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"]
Oct 30 19:46:12.035003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.034977    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bb9c3-gkrjf"]
Oct 30 19:46:12.099576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.099544    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23511b6-5c7c-434f-9879-9ee98f3312d7" path="/var/lib/kubelet/pods/a23511b6-5c7c-434f-9879-9ee98f3312d7/volumes"
Oct 30 19:46:12.939889 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.939860    4320 generic.go:356] "Generic (PLEG): container finished" podID="492bf6de-78cb-4039-b875-a8b337cb255f" containerID="17b6536431f59aa60460c1ff3c627d750050747a4c7f4a9682b762acd9480426" exitCode=0
Oct 30 19:46:12.940309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.939942    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8" event={"ID":"492bf6de-78cb-4039-b875-a8b337cb255f","Type":"ContainerDied","Data":"17b6536431f59aa60460c1ff3c627d750050747a4c7f4a9682b762acd9480426"}
Oct 30 19:46:12.940309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:12.940008    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8" event={"ID":"492bf6de-78cb-4039-b875-a8b337cb255f","Type":"ContainerStarted","Data":"0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7"}
Oct 30 19:46:13.563152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:13.563107    4320 scope.go:122] "RemoveContainer" containerID="2cfd2db527f9919ac5779ca2d57b2246d8a350a357d9f7cc39186010822a5d49"
Oct 30 19:46:14.110642 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.110616    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:14.203940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.203853    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\" (UniqueName: \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\") pod \"492bf6de-78cb-4039-b875-a8b337cb255f\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") "
Oct 30 19:46:14.204144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.204127    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\") pod \"492bf6de-78cb-4039-b875-a8b337cb255f\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") "
Oct 30 19:46:14.204217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.204186    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\") pod \"492bf6de-78cb-4039-b875-a8b337cb255f\" (UID: \"492bf6de-78cb-4039-b875-a8b337cb255f\") "
Oct 30 19:46:14.204567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.204540    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca" pod "492bf6de-78cb-4039-b875-a8b337cb255f" (UID: "492bf6de-78cb-4039-b875-a8b337cb255f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:14.204935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.204917    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:14.206092 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.206065    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g" pod "492bf6de-78cb-4039-b875-a8b337cb255f" (UID: "492bf6de-78cb-4039-b875-a8b337cb255f"). InnerVolumeSpecName "kube-api-access-vch6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:14.206182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.206112    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca" pod "492bf6de-78cb-4039-b875-a8b337cb255f" (UID: "492bf6de-78cb-4039-b875-a8b337cb255f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:14.305845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.305805    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492bf6de-78cb-4039-b875-a8b337cb255f-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:14.305845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.305831    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vch6g\" (UniqueName: \"kubernetes.io/projected/492bf6de-78cb-4039-b875-a8b337cb255f-kube-api-access-vch6g\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:14.955300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.955264    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8" event={"ID":"492bf6de-78cb-4039-b875-a8b337cb255f","Type":"ContainerDied","Data":"0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7"}
Oct 30 19:46:14.955300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.955298    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"
Oct 30 19:46:14.955499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:14.955312    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bbe76aa1d694a0ee755051307feb9166c293c7bef279c472180c257b35b3ab7"
Oct 30 19:46:17.410990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:17.410959    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:17.435970 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:17.435942    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:17.987801 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:17.987768    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:46:19.699336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.699301    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"]
Oct 30 19:46:19.700117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.700098    4320 cpu_manager.go:414] "containerMap: removing container" podUID="492bf6de-78cb-4039-b875-a8b337cb255f" containerName="revoke-keys"
Oct 30 19:46:19.700165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.700120    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492bf6de-78cb-4039-b875-a8b337cb255f" containerName="revoke-keys"
Oct 30 19:46:19.700284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.700272    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="492bf6de-78cb-4039-b875-a8b337cb255f" containerName="revoke-keys"
Oct 30 19:46:19.704173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.704155    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.723383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.723351    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"]
Oct 30 19:46:19.746629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.746589    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:19.746863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.746843    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:19.787759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.787731    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.787861 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.787801    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qzd\" (UniqueName: \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.788020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.787997    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.889814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.889785    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qzd\" (UniqueName: \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.889988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.889951    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.890146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.890130    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.890790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.890767    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.892276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.892250    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:19.908848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:19.908821    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qzd\" (UniqueName: \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\") pod \"maas-api-revoke-sub-92bfd1f1-cx7g6\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:20.019396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:20.019321    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:20.328444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:20.328417    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"]
Oct 30 19:46:20.329987 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:20.329959    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037b5c20_c7fb_4e9d_b75a_753a404b6cee.slice/crio-1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255 WatchSource:0}: Error finding container 1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255: Status 404 returned error can't find the container with id 1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255
Oct 30 19:46:21.008545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:21.008505    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6" event={"ID":"037b5c20-c7fb-4e9d-b75a-753a404b6cee","Type":"ContainerStarted","Data":"a2ec8de629585ec08f54e02d59475975d875626f46d6bc9b3069042dcf6993a5"}
Oct 30 19:46:21.009073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:21.008558    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6" event={"ID":"037b5c20-c7fb-4e9d-b75a-753a404b6cee","Type":"ContainerStarted","Data":"1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255"}
Oct 30 19:46:21.049689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:21.049624    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6" podStartSLOduration=2.049591412 podStartE2EDuration="2.049591412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:21.04571565 +0000 UTC m=+2529.081228126" watchObservedRunningTime="2026-10-30 19:46:21.049591412 +0000 UTC m=+2529.085103879"
Oct 30 19:46:22.017521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:22.017480    4320 generic.go:356] "Generic (PLEG): container finished" podID="037b5c20-c7fb-4e9d-b75a-753a404b6cee" containerID="a2ec8de629585ec08f54e02d59475975d875626f46d6bc9b3069042dcf6993a5" exitCode=0
Oct 30 19:46:22.018024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:22.017569    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6" event={"ID":"037b5c20-c7fb-4e9d-b75a-753a404b6cee","Type":"ContainerDied","Data":"a2ec8de629585ec08f54e02d59475975d875626f46d6bc9b3069042dcf6993a5"}
Oct 30 19:46:22.801620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:22.785408    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:46:22.801620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:22.785696    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy" containerID="cri-o://868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857" gracePeriod=30
Oct 30 19:46:23.188058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.188031    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:23.352592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.352508    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\" (UniqueName: \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\") pod \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") "
Oct 30 19:46:23.352592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.352570    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\") pod \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") "
Oct 30 19:46:23.352924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.352763    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\") pod \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\" (UID: \"037b5c20-c7fb-4e9d-b75a-753a404b6cee\") "
Oct 30 19:46:23.353875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.353301    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca" pod "037b5c20-c7fb-4e9d-b75a-753a404b6cee" (UID: "037b5c20-c7fb-4e9d-b75a-753a404b6cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:23.353875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.353572    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.354918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.354889    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca" pod "037b5c20-c7fb-4e9d-b75a-753a404b6cee" (UID: "037b5c20-c7fb-4e9d-b75a-753a404b6cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:23.355142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.355124    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd" pod "037b5c20-c7fb-4e9d-b75a-753a404b6cee" (UID: "037b5c20-c7fb-4e9d-b75a-753a404b6cee"). InnerVolumeSpecName "kube-api-access-l6qzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:23.455447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.455406    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qzd\" (UniqueName: \"kubernetes.io/projected/037b5c20-c7fb-4e9d-b75a-753a404b6cee-kube-api-access-l6qzd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.455447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:23.455446    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/037b5c20-c7fb-4e9d-b75a-753a404b6cee-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:24.037301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:24.036763    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"
Oct 30 19:46:24.037301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:24.036793    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6" event={"ID":"037b5c20-c7fb-4e9d-b75a-753a404b6cee","Type":"ContainerDied","Data":"1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255"}
Oct 30 19:46:24.037301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:24.036855    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de76b708b6b8e50505f826a23bdbd67659b34b46ccf77007085c678e198b255"
Oct 30 19:46:27.584620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:27.583758    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:46:28.060696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.060674    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:46:28.068955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.068929    4320 generic.go:356] "Generic (PLEG): container finished" podID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerID="868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857" exitCode=0
Oct 30 19:46:28.069090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.068972    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"
Oct 30 19:46:28.069090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.069006    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerDied","Data":"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"}
Oct 30 19:46:28.069090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.069065    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn" event={"ID":"15545d8b-27bf-4756-86e0-ce9a7d4b82d1","Type":"ContainerDied","Data":"8b271fc223f84e5f18c3ee58a9dcb00595fdf1779ef847faee23aa80d7888aa4"}
Oct 30 19:46:28.069221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.069089    4320 scope.go:122] "RemoveContainer" containerID="868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"
Oct 30 19:46:28.079364 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.079346    4320 scope.go:122] "RemoveContainer" containerID="2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"
Oct 30 19:46:28.088581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.088557    4320 scope.go:122] "RemoveContainer" containerID="868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"
Oct 30 19:46:28.088894 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:46:28.088872    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857\": container with ID starting with 868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857 not found: ID does not exist" containerID="868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"
Oct 30 19:46:28.088999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.088901    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857"} err="failed to get container status \"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857\": rpc error: code = NotFound desc = could not find container \"868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857\": container with ID starting with 868b15dedf020738b059ada40e88b118aefc3538cde9d7e440cffbe35f4e8857 not found: ID does not exist"
Oct 30 19:46:28.088999 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.088917    4320 scope.go:122] "RemoveContainer" containerID="2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"
Oct 30 19:46:28.089176 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:46:28.089156    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e\": container with ID starting with 2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e not found: ID does not exist" containerID="2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"
Oct 30 19:46:28.089275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.089180    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e"} err="failed to get container status \"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e\": rpc error: code = NotFound desc = could not find container \"2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e\": container with ID starting with 2cb3694a970860f07b923197a7d687c8590139c94f6b57521df35db5fed7407e not found: ID does not exist"
Oct 30 19:46:28.244326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244251    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244286    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244337    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244445    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244487    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244533    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244577 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244543    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:28.244885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244653    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244771    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.244885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244658    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:28.244885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244740    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:28.244885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244813    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\") pod \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\" (UID: \"15545d8b-27bf-4756-86e0-ce9a7d4b82d1\") "
Oct 30 19:46:28.245149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244905    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:28.245149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.244782    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:28.245630 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.245587    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.245761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.245634    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.245761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.245651    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.245761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.245665    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.245761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.245680    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.246914 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.246885    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:28.247029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.246930    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:28.247029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.246946    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:28.247029 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.246985    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt" pod "15545d8b-27bf-4756-86e0-ce9a7d4b82d1" (UID: "15545d8b-27bf-4756-86e0-ce9a7d4b82d1"). InnerVolumeSpecName "kube-api-access-qn2bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:28.347459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.347430    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.347459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.347457    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn2bt\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-kube-api-access-qn2bt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.347666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.347468    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.347666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.347477    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15545d8b-27bf-4756-86e0-ce9a7d4b82d1-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:28.419975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.419950    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:46:28.431445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:28.431421    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-cnvpn"]
Oct 30 19:46:28.646037 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:46:28.645997    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:46:30.055285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:30.054125    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:46:30.114502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:30.114464    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" path="/var/lib/kubelet/pods/15545d8b-27bf-4756-86e0-ce9a7d4b82d1/volumes"
Oct 30 19:46:32.660962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.660926    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"]
Oct 30 19:46:32.661587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661572    4320 cpu_manager.go:414] "containerMap: removing container" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661587    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661614    4320 cpu_manager.go:414] "containerMap: removing container" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661619    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661655    4320 cpu_manager.go:414] "containerMap: removing container" podUID="037b5c20-c7fb-4e9d-b75a-753a404b6cee" containerName="revoke-keys"
Oct 30 19:46:32.661718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661660    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037b5c20-c7fb-4e9d-b75a-753a404b6cee" containerName="revoke-keys"
Oct 30 19:46:32.661851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661768    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661776    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="15545d8b-27bf-4756-86e0-ce9a7d4b82d1" containerName="istio-proxy"
Oct 30 19:46:32.661851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.661790    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="037b5c20-c7fb-4e9d-b75a-753a404b6cee" containerName="revoke-keys"
Oct 30 19:46:32.667020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.667003    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.683609 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.683568    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"]
Oct 30 19:46:32.689815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.689793    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:32.690172 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.690117    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:32.773510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.773473    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.773685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.773617    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n95x\" (UniqueName: \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.773893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.773865    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.875320 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.875285    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.875493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.875452    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.875550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.875516    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n95x\" (UniqueName: \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.876149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.876116    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.878108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.878076    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:32.894464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:32.894430    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n95x\" (UniqueName: \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\") pod \"maas-api-revoke-sub-12e402df-6s5fn\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:33.000958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:33.000822    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:33.310472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:33.310386    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"]
Oct 30 19:46:33.323451 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:33.323421    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6d52d7_698b_48ae_89e8_53ab9c061af9.slice/crio-e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1 WatchSource:0}: Error finding container e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1: Status 404 returned error can't find the container with id e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1
Oct 30 19:46:34.166705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:34.166669    4320 generic.go:356] "Generic (PLEG): container finished" podID="3e6d52d7-698b-48ae-89e8-53ab9c061af9" containerID="376af8768db6c9e7a76c09e7ad5c13b405b61cc97b4a741fd1b801fde537af81" exitCode=0
Oct 30 19:46:34.167082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:34.166742    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn" event={"ID":"3e6d52d7-698b-48ae-89e8-53ab9c061af9","Type":"ContainerDied","Data":"376af8768db6c9e7a76c09e7ad5c13b405b61cc97b4a741fd1b801fde537af81"}
Oct 30 19:46:34.167082 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:34.166799    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn" event={"ID":"3e6d52d7-698b-48ae-89e8-53ab9c061af9","Type":"ContainerStarted","Data":"e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1"}
Oct 30 19:46:35.347225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.347197    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:35.408881 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.408849    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\") pod \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") "
Oct 30 19:46:35.409047 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.408935    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\" (UniqueName: \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\") pod \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") "
Oct 30 19:46:35.409127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.409061    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\") pod \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\" (UID: \"3e6d52d7-698b-48ae-89e8-53ab9c061af9\") "
Oct 30 19:46:35.409271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.409242    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca" pod "3e6d52d7-698b-48ae-89e8-53ab9c061af9" (UID: "3e6d52d7-698b-48ae-89e8-53ab9c061af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:35.409786 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.409760    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:35.411924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.411894    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca" pod "3e6d52d7-698b-48ae-89e8-53ab9c061af9" (UID: "3e6d52d7-698b-48ae-89e8-53ab9c061af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:35.412048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.411944    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x" pod "3e6d52d7-698b-48ae-89e8-53ab9c061af9" (UID: "3e6d52d7-698b-48ae-89e8-53ab9c061af9"). InnerVolumeSpecName "kube-api-access-9n95x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:35.511510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.511451    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n95x\" (UniqueName: \"kubernetes.io/projected/3e6d52d7-698b-48ae-89e8-53ab9c061af9-kube-api-access-9n95x\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:35.511510 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:35.511475    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6d52d7-698b-48ae-89e8-53ab9c061af9-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:36.214772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.214727    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn" event={"ID":"3e6d52d7-698b-48ae-89e8-53ab9c061af9","Type":"ContainerDied","Data":"e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1"}
Oct 30 19:46:36.215030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.215014    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e3326df6084c912aff0c492cbdf7fbaed5d26c4a4be8deff2904e11094a6f1"
Oct 30 19:46:36.215203 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.215180    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"
Oct 30 19:46:36.543260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.543187    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"]
Oct 30 19:46:36.543866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.543850    4320 cpu_manager.go:414] "containerMap: removing container" podUID="3e6d52d7-698b-48ae-89e8-53ab9c061af9" containerName="revoke-keys"
Oct 30 19:46:36.543926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.543868    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6d52d7-698b-48ae-89e8-53ab9c061af9" containerName="revoke-keys"
Oct 30 19:46:36.544021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.544011    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6d52d7-698b-48ae-89e8-53ab9c061af9" containerName="revoke-keys"
Oct 30 19:46:36.547747 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.547726    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.562139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.562118    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:36.562512 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.562495    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:36.568366 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.568332    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"]
Oct 30 19:46:36.642406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.642370    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.642656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.642639    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.642753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.642701    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cks\" (UniqueName: \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.746067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.746028    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.746233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.746152    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cks\" (UniqueName: \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.746281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.746237    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.746975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.746949    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.748457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.748438    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.768377 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.768353    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cks\" (UniqueName: \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\") pod \"maas-api-revoke-sub-241aecd4-56pqn\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:36.863987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:36.863911    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:37.169643 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:37.169606    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"]
Oct 30 19:46:37.170045 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:37.170022    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e2b82f_8f2d_427b_b27d_81cba5a6cc4e.slice/crio-811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367 WatchSource:0}: Error finding container 811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367: Status 404 returned error can't find the container with id 811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367
Oct 30 19:46:37.223563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:37.223538    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn" event={"ID":"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e","Type":"ContainerStarted","Data":"811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367"}
Oct 30 19:46:38.231960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:38.231932    4320 generic.go:356] "Generic (PLEG): container finished" podID="b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" containerID="8cb088c85834326e3d5e460fb47ddfea31f6b218f98e9f8ea8a749f1ded23db4" exitCode=0
Oct 30 19:46:38.232306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:38.232023    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn" event={"ID":"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e","Type":"ContainerDied","Data":"8cb088c85834326e3d5e460fb47ddfea31f6b218f98e9f8ea8a749f1ded23db4"}
Oct 30 19:46:39.394080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.394049    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:39.481570 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.481542    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\") pod \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") "
Oct 30 19:46:39.481768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.481684    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\") pod \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") "
Oct 30 19:46:39.481921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.481905    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\" (UniqueName: \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\") pod \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\" (UID: \"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e\") "
Oct 30 19:46:39.481986 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.481940    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca" pod "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" (UID: "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:39.482738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.482710    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:39.484018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.483988    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca" pod "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" (UID: "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:39.485093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.485063    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks" pod "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" (UID: "b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e"). InnerVolumeSpecName "kube-api-access-p9cks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:39.583952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.583927    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cks\" (UniqueName: \"kubernetes.io/projected/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-kube-api-access-p9cks\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:39.583952 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:39.583949    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:40.248276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:40.248244    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"
Oct 30 19:46:40.248437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:40.248281    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn" event={"ID":"b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e","Type":"ContainerDied","Data":"811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367"}
Oct 30 19:46:40.248437 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:40.248334    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="811665e3cae19e23f7f0bb4121e4edd6077be911c0c8d4824a211efea10e0367"
Oct 30 19:46:43.011350 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.011320    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"]
Oct 30 19:46:43.011988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.011973    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" containerName="revoke-keys"
Oct 30 19:46:43.012044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.011988    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" containerName="revoke-keys"
Oct 30 19:46:43.012118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.012109    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" containerName="revoke-keys"
Oct 30 19:46:43.015628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.015610    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.034498 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.034468    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"]
Oct 30 19:46:43.041014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.040990    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:43.041143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.041054    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:43.128944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.128919    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8lp\" (UniqueName: \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.129097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.128975    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.129097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.129068    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.230161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.230136    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8lp\" (UniqueName: \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.230502 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.230476    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.230592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.230567    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.231196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.231171    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.232964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.232936    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.249234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.249211    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8lp\" (UniqueName: \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\") pod \"maas-api-revoke-sub-9d9cbea9-7wqbp\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.329466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.329401    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:43.711853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:43.710459    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"]
Oct 30 19:46:44.280273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:44.280236    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp" event={"ID":"32663bd8-b235-4b24-880b-8ce0081fb228","Type":"ContainerStarted","Data":"dde9bb00783832b348ac3f61545397a5c154bbc38cddd7ea6fed64c709127271"}
Oct 30 19:46:44.280765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:44.280284    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp" event={"ID":"32663bd8-b235-4b24-880b-8ce0081fb228","Type":"ContainerStarted","Data":"120ed38e86188ea7366c5ac6050d73fee308bbb3f15ad95d41732c7f972d1dd2"}
Oct 30 19:46:44.321988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:44.321939    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp" podStartSLOduration=2.321923728 podStartE2EDuration="2.321923728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.317056025 +0000 UTC m=+2552.352568488" watchObservedRunningTime="2026-10-30 19:46:44.321923728 +0000 UTC m=+2552.357436191"
Oct 30 19:46:45.082978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.082873    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" podUID="60924baf-451e-417c-98b9-3fda824024d2" containerName="main" containerID="cri-o://46478be1918dec4390da6e399ad59181d3775077f7cf283aa9deb850095a4d8c" gracePeriod=45
Oct 30 19:46:45.337738 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.337647    4320 generic.go:356] "Generic (PLEG): container finished" podID="60924baf-451e-417c-98b9-3fda824024d2" containerID="46478be1918dec4390da6e399ad59181d3775077f7cf283aa9deb850095a4d8c" exitCode=0
Oct 30 19:46:45.338400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.337731    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" event={"ID":"60924baf-451e-417c-98b9-3fda824024d2","Type":"ContainerDied","Data":"46478be1918dec4390da6e399ad59181d3775077f7cf283aa9deb850095a4d8c"}
Oct 30 19:46:45.340413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.340387    4320 generic.go:356] "Generic (PLEG): container finished" podID="32663bd8-b235-4b24-880b-8ce0081fb228" containerID="dde9bb00783832b348ac3f61545397a5c154bbc38cddd7ea6fed64c709127271" exitCode=0
Oct 30 19:46:45.340579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.340561    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp" event={"ID":"32663bd8-b235-4b24-880b-8ce0081fb228","Type":"ContainerDied","Data":"dde9bb00783832b348ac3f61545397a5c154bbc38cddd7ea6fed64c709127271"}
Oct 30 19:46:45.397458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.397429    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:46:45.471842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.471803    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472010 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.471948    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472011    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472053    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:45.472075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472070    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472236 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472168    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\" (UniqueName: \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472235    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\") pod \"60924baf-451e-417c-98b9-3fda824024d2\" (UID: \"60924baf-451e-417c-98b9-3fda824024d2\") "
Oct 30 19:46:45.472797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.472762    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:45.473061 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.473034    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:45.473186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.473066    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:45.474585 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.474562    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "kube-api-access-h7b74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:45.474850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.474823    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:45.475721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.475697    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:45.475995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.475971    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm" pod "60924baf-451e-417c-98b9-3fda824024d2" (UID: "60924baf-451e-417c-98b9-3fda824024d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:45.574664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.574639    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:45.574767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.574665    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60924baf-451e-417c-98b9-3fda824024d2-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:45.574767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.574679    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60924baf-451e-417c-98b9-3fda824024d2-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:45.574767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:45.574691    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7b74\" (UniqueName: \"kubernetes.io/projected/60924baf-451e-417c-98b9-3fda824024d2-kube-api-access-h7b74\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:46.350511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.350482    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"
Oct 30 19:46:46.350998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.350512    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj" event={"ID":"60924baf-451e-417c-98b9-3fda824024d2","Type":"ContainerDied","Data":"108e58d377dd4bae369d604fb206ea7a7e76ff4016d450959eaba5fd26dd6fba"}
Oct 30 19:46:46.350998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.350562    4320 scope.go:122] "RemoveContainer" containerID="46478be1918dec4390da6e399ad59181d3775077f7cf283aa9deb850095a4d8c"
Oct 30 19:46:46.405668 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.405640    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:46:46.424307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.424279    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d9eb76/auth-test-model-d9eb76-kserve-66b68f58cc-jkjwj"]
Oct 30 19:46:46.527938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.527914    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:46.692426 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.692396    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\" (UniqueName: \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\") pod \"32663bd8-b235-4b24-880b-8ce0081fb228\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") "
Oct 30 19:46:46.692629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.692587    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\") pod \"32663bd8-b235-4b24-880b-8ce0081fb228\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") "
Oct 30 19:46:46.692694 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.692666    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\") pod \"32663bd8-b235-4b24-880b-8ce0081fb228\" (UID: \"32663bd8-b235-4b24-880b-8ce0081fb228\") "
Oct 30 19:46:46.693110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.693075    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca" pod "32663bd8-b235-4b24-880b-8ce0081fb228" (UID: "32663bd8-b235-4b24-880b-8ce0081fb228"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:46.693406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.693384    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:46.694659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.694634    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca" pod "32663bd8-b235-4b24-880b-8ce0081fb228" (UID: "32663bd8-b235-4b24-880b-8ce0081fb228"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:46.694785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.694707    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp" pod "32663bd8-b235-4b24-880b-8ce0081fb228" (UID: "32663bd8-b235-4b24-880b-8ce0081fb228"). InnerVolumeSpecName "kube-api-access-kh8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:46.795075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.795052    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh8lp\" (UniqueName: \"kubernetes.io/projected/32663bd8-b235-4b24-880b-8ce0081fb228-kube-api-access-kh8lp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:46.795185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:46.795079    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32663bd8-b235-4b24-880b-8ce0081fb228-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:47.362765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:47.362730    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp" event={"ID":"32663bd8-b235-4b24-880b-8ce0081fb228","Type":"ContainerDied","Data":"120ed38e86188ea7366c5ac6050d73fee308bbb3f15ad95d41732c7f972d1dd2"}
Oct 30 19:46:47.362765 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:47.362748    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"
Oct 30 19:46:47.363210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:47.362787    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120ed38e86188ea7366c5ac6050d73fee308bbb3f15ad95d41732c7f972d1dd2"
Oct 30 19:46:48.098543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:48.098507    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60924baf-451e-417c-98b9-3fda824024d2" path="/var/lib/kubelet/pods/60924baf-451e-417c-98b9-3fda824024d2/volumes"
Oct 30 19:46:50.512622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.511455    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"]
Oct 30 19:46:50.512622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.512493    4320 cpu_manager.go:414] "containerMap: removing container" podUID="60924baf-451e-417c-98b9-3fda824024d2" containerName="main"
Oct 30 19:46:50.512622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.512525    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60924baf-451e-417c-98b9-3fda824024d2" containerName="main"
Oct 30 19:46:50.513273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.512653    4320 cpu_manager.go:414] "containerMap: removing container" podUID="32663bd8-b235-4b24-880b-8ce0081fb228" containerName="revoke-keys"
Oct 30 19:46:50.513273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.512664    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32663bd8-b235-4b24-880b-8ce0081fb228" containerName="revoke-keys"
Oct 30 19:46:50.513273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.513160    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="32663bd8-b235-4b24-880b-8ce0081fb228" containerName="revoke-keys"
Oct 30 19:46:50.513273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.513209    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="60924baf-451e-417c-98b9-3fda824024d2" containerName="main"
Oct 30 19:46:50.519410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.518785    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.529564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.528655    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"]
Oct 30 19:46:50.535436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.535416    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:46:50.535539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.535479    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:50.657799 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.657690    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.657989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.657900    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.658146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.658112    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnf5t\" (UniqueName: \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.759815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.759779    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnf5t\" (UniqueName: \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.759963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.759836    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.759963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.759945    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.760605 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.760577    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.762226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.762208    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.779409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.779350    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnf5t\" (UniqueName: \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\") pod \"maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:50.843126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:50.843101    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:51.154971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:51.154934    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"]
Oct 30 19:46:51.163343 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:51.163305    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73009d05_be46_434e_913c_ec64b0ebf067.slice/crio-2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929 WatchSource:0}: Error finding container 2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929: Status 404 returned error can't find the container with id 2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929
Oct 30 19:46:51.455934 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:51.455851    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld" event={"ID":"73009d05-be46-434e-913c-ec64b0ebf067","Type":"ContainerStarted","Data":"2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929"}
Oct 30 19:46:52.465481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:52.465443    4320 generic.go:356] "Generic (PLEG): container finished" podID="73009d05-be46-434e-913c-ec64b0ebf067" containerID="d314c6888cb06c8b2f05dfe81130488120b3dbe20e98cf319a7851fb4f54c422" exitCode=0
Oct 30 19:46:52.465908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:52.465525    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld" event={"ID":"73009d05-be46-434e-913c-ec64b0ebf067","Type":"ContainerDied","Data":"d314c6888cb06c8b2f05dfe81130488120b3dbe20e98cf319a7851fb4f54c422"}
Oct 30 19:46:53.648217 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.648191    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:53.703700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.703673    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\") pod \"73009d05-be46-434e-913c-ec64b0ebf067\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") "
Oct 30 19:46:53.703864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.703845    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\") pod \"73009d05-be46-434e-913c-ec64b0ebf067\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") "
Oct 30 19:46:53.703907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.703885    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\" (UniqueName: \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\") pod \"73009d05-be46-434e-913c-ec64b0ebf067\" (UID: \"73009d05-be46-434e-913c-ec64b0ebf067\") "
Oct 30 19:46:53.704093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.704071    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca" pod "73009d05-be46-434e-913c-ec64b0ebf067" (UID: "73009d05-be46-434e-913c-ec64b0ebf067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:53.704440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.704411    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:53.705980 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.705955    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca" pod "73009d05-be46-434e-913c-ec64b0ebf067" (UID: "73009d05-be46-434e-913c-ec64b0ebf067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:53.706127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.706110    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t" pod "73009d05-be46-434e-913c-ec64b0ebf067" (UID: "73009d05-be46-434e-913c-ec64b0ebf067"). InnerVolumeSpecName "kube-api-access-gnf5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:53.806206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.806152    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73009d05-be46-434e-913c-ec64b0ebf067-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:53.806206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:53.806177    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnf5t\" (UniqueName: \"kubernetes.io/projected/73009d05-be46-434e-913c-ec64b0ebf067-kube-api-access-gnf5t\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.025500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.025469    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"]
Oct 30 19:46:54.026199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.026183    4320 cpu_manager.go:414] "containerMap: removing container" podUID="73009d05-be46-434e-913c-ec64b0ebf067" containerName="revoke-keys"
Oct 30 19:46:54.026262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.026200    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73009d05-be46-434e-913c-ec64b0ebf067" containerName="revoke-keys"
Oct 30 19:46:54.026308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.026299    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="73009d05-be46-434e-913c-ec64b0ebf067" containerName="revoke-keys"
Oct 30 19:46:54.030028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.030006    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.048294 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.048271    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"]
Oct 30 19:46:54.108979 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.108871    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48vp\" (UniqueName: \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.109162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.109084    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.109162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.109112    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.210740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.210713    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.210740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.210743    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.210907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.210835    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48vp\" (UniqueName: \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.211448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.211423    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.212949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.212929    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.229169 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.229148    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48vp\" (UniqueName: \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\") pod \"maas-api-revoke-sub-3caa79e2-x6hfz\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.348008 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.347984    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:54.504206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.504170    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"
Oct 30 19:46:54.504402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.504213    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld" event={"ID":"73009d05-be46-434e-913c-ec64b0ebf067","Type":"ContainerDied","Data":"2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929"}
Oct 30 19:46:54.504402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.504277    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e70072300996447989dea49b662edfc8d9339fbe5a2ab1f47a00e5be35dd929"
Oct 30 19:46:54.651087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:54.651058    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"]
Oct 30 19:46:54.652043 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:46:54.652018    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103f160e_141a_4d5e_8f7e_2a005e5343e0.slice/crio-d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5 WatchSource:0}: Error finding container d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5: Status 404 returned error can't find the container with id d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5
Oct 30 19:46:55.513356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:55.513321    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz" event={"ID":"103f160e-141a-4d5e-8f7e-2a005e5343e0","Type":"ContainerStarted","Data":"b0543e4c734b85ee0e4c3061cbe5b2bb2614f2d8b6e3d599f07f935c363c78ac"}
Oct 30 19:46:55.513539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:55.513386    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz" event={"ID":"103f160e-141a-4d5e-8f7e-2a005e5343e0","Type":"ContainerStarted","Data":"d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5"}
Oct 30 19:46:55.553515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:55.553467    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz" podStartSLOduration=2.553454679 podStartE2EDuration="2.553454679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:55.55143334 +0000 UTC m=+2563.586945805" watchObservedRunningTime="2026-10-30 19:46:55.553454679 +0000 UTC m=+2563.588967182"
Oct 30 19:46:56.522514 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:56.522481    4320 generic.go:356] "Generic (PLEG): container finished" podID="103f160e-141a-4d5e-8f7e-2a005e5343e0" containerID="b0543e4c734b85ee0e4c3061cbe5b2bb2614f2d8b6e3d599f07f935c363c78ac" exitCode=0
Oct 30 19:46:56.522912 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:56.522559    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz" event={"ID":"103f160e-141a-4d5e-8f7e-2a005e5343e0","Type":"ContainerDied","Data":"b0543e4c734b85ee0e4c3061cbe5b2bb2614f2d8b6e3d599f07f935c363c78ac"}
Oct 30 19:46:57.702671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.702650    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:57.756205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.756170    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\" (UniqueName: \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\") pod \"103f160e-141a-4d5e-8f7e-2a005e5343e0\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") "
Oct 30 19:46:57.756359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.756344    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\") pod \"103f160e-141a-4d5e-8f7e-2a005e5343e0\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") "
Oct 30 19:46:57.756720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.756693    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\") pod \"103f160e-141a-4d5e-8f7e-2a005e5343e0\" (UID: \"103f160e-141a-4d5e-8f7e-2a005e5343e0\") "
Oct 30 19:46:57.757107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.757057    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca" pod "103f160e-141a-4d5e-8f7e-2a005e5343e0" (UID: "103f160e-141a-4d5e-8f7e-2a005e5343e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:57.757870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.757841    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:57.759130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.759103    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp" pod "103f160e-141a-4d5e-8f7e-2a005e5343e0" (UID: "103f160e-141a-4d5e-8f7e-2a005e5343e0"). InnerVolumeSpecName "kube-api-access-m48vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:57.759788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.759760    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca" pod "103f160e-141a-4d5e-8f7e-2a005e5343e0" (UID: "103f160e-141a-4d5e-8f7e-2a005e5343e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:57.859723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.859668    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103f160e-141a-4d5e-8f7e-2a005e5343e0-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:57.859723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:57.859692    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m48vp\" (UniqueName: \"kubernetes.io/projected/103f160e-141a-4d5e-8f7e-2a005e5343e0-kube-api-access-m48vp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:46:58.544891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:58.544855    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"
Oct 30 19:46:58.544891 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:58.544876    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz" event={"ID":"103f160e-141a-4d5e-8f7e-2a005e5343e0","Type":"ContainerDied","Data":"d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5"}
Oct 30 19:46:58.545226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:46:58.544941    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d465f348de9c86bcd6cce7aea7c2bdbbdf09290cca49cd141d439955ea57c8d5"
Oct 30 19:46:58.649924 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:46:58.649889    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:47:04.549249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.549211    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:47:04.549863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.549846    4320 cpu_manager.go:414] "containerMap: removing container" podUID="103f160e-141a-4d5e-8f7e-2a005e5343e0" containerName="revoke-keys"
Oct 30 19:47:04.549863 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.549862    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103f160e-141a-4d5e-8f7e-2a005e5343e0" containerName="revoke-keys"
Oct 30 19:47:04.549992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.549982    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="103f160e-141a-4d5e-8f7e-2a005e5343e0" containerName="revoke-keys"
Oct 30 19:47:04.553817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.553795    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.575460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.575433    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:47:04.581504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.581478    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"default-dockercfg-d5dwr\""
Oct 30 19:47:04.581504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.581499    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"sub-test-model-6b7ccc-kserve-self-signed-certs\""
Oct 30 19:47:04.582318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.582293    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"openshift-service-ca.crt\""
Oct 30 19:47:04.582509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.582489    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b7ccc\"/\"kube-root-ca.crt\""
Oct 30 19:47:04.609534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.609503    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.609758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.609734    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.609923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.609901    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.610004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.609936    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.610004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.609971    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.610116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.610018    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fxg\" (UniqueName: \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712165 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712117    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712268    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712296    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712335    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712392    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fxg\" (UniqueName: \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712547 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712507    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712540    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712704    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.712849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.712708    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.714860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.714831    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.715001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.714989    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.731832 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.731804    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fxg\" (UniqueName: \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\") pod \"sub-test-model-6b7ccc-kserve-8656454977-w9vdc\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") " pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:04.900620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:04.899206    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:05.275788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:05.275753    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:47:05.276033 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:47:05.276012    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod204d1aac_7a6c_4950_9632_86ea45fb8f48.slice/crio-5d7f72e574fda6c96deb4fcc45dc2cd4750b937589be053a12736f2a87249dda WatchSource:0}: Error finding container 5d7f72e574fda6c96deb4fcc45dc2cd4750b937589be053a12736f2a87249dda: Status 404 returned error can't find the container with id 5d7f72e574fda6c96deb4fcc45dc2cd4750b937589be053a12736f2a87249dda
Oct 30 19:47:05.632896 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:05.632862    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" event={"ID":"204d1aac-7a6c-4950-9632-86ea45fb8f48","Type":"ContainerStarted","Data":"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"}
Oct 30 19:47:05.633262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:05.632911    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" event={"ID":"204d1aac-7a6c-4950-9632-86ea45fb8f48","Type":"ContainerStarted","Data":"5d7f72e574fda6c96deb4fcc45dc2cd4750b937589be053a12736f2a87249dda"}
Oct 30 19:47:05.674504 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:05.674457    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" podStartSLOduration=1.6744454690000001 podStartE2EDuration="1.674445469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:05.670806552 +0000 UTC m=+2573.706319019" watchObservedRunningTime="2026-10-30 19:47:05.674445469 +0000 UTC m=+2573.709957945"
Oct 30 19:47:05.903840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:05.903760    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:13.655538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.655506    4320 scope.go:122] "RemoveContainer" containerID="3e629c83e16aa46be071111cfaea21597b124e5f83dc228ff2ef55ad1b5b5db7"
Oct 30 19:47:13.666318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.666294    4320 scope.go:122] "RemoveContainer" containerID="8f09fef76a04ace716346bdf0b0c24a013a5cec363ab855feb8c0edd9d594377"
Oct 30 19:47:13.675690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.675672    4320 scope.go:122] "RemoveContainer" containerID="57411a186c44a59f5a31b68078944fca7fbebd90c57af1d240458bb359d50049"
Oct 30 19:47:13.684987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.684967    4320 scope.go:122] "RemoveContainer" containerID="3c301db8b06d7096f5144aded92e8a7e07f28bf3ed9d02059b0dc78238e73228"
Oct 30 19:47:13.694317 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.694293    4320 scope.go:122] "RemoveContainer" containerID="79e53a1470706830e3fab7fe74c8205335274deb1cb59f39d2dd81e2962e5d63"
Oct 30 19:47:13.703505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:13.703482    4320 scope.go:122] "RemoveContainer" containerID="dac7d6a9ffba124f592d63ad863fc1f6a33c9c8b686263791255e16bb410fdad"
Oct 30 19:47:14.904062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:14.904021    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:14.917370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:14.917348    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:15.719901 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:15.719862    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:47:22.838876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:22.838840    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"]
Oct 30 19:47:22.849561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:22.849534    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-08bcfe-5jsld"]
Oct 30 19:47:24.034478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.034446    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"]
Oct 30 19:47:24.044360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.044332    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.050569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.050539    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"]
Oct 30 19:47:24.101442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.101410    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:24.102291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.102262    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:47:24.111065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.111035    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73009d05-be46-434e-913c-ec64b0ebf067" path="/var/lib/kubelet/pods/73009d05-be46-434e-913c-ec64b0ebf067/volumes"
Oct 30 19:47:24.143153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.140639    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.143153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.140845    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wzt\" (UniqueName: \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.143153 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.141063    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.243523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.243485    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.243718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.243663    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wzt\" (UniqueName: \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.243851 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.243832    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.244704 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.244677    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.246392 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.246356    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.265388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.265358    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wzt\" (UniqueName: \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\") pod \"maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.370792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.370736    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:24.691987 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:47:24.691942    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f7532f_d80e_4ed2_a01a_1cfb804460fe.slice/crio-9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292 WatchSource:0}: Error finding container 9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292: Status 404 returned error can't find the container with id 9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292
Oct 30 19:47:24.692188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.692168    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"]
Oct 30 19:47:24.782361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:24.782326    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8" event={"ID":"c9f7532f-d80e-4ed2-a01a-1cfb804460fe","Type":"ContainerStarted","Data":"9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292"}
Oct 30 19:47:25.792544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:25.792517    4320 generic.go:356] "Generic (PLEG): container finished" podID="c9f7532f-d80e-4ed2-a01a-1cfb804460fe" containerID="6e4470c40b62ab0163d33131c01b4f0516be2eaae61f7f3a0071e4279d6c979b" exitCode=0
Oct 30 19:47:25.792939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:25.792638    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8" event={"ID":"c9f7532f-d80e-4ed2-a01a-1cfb804460fe","Type":"ContainerDied","Data":"6e4470c40b62ab0163d33131c01b4f0516be2eaae61f7f3a0071e4279d6c979b"}
Oct 30 19:47:27.001792 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.001768    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:27.092251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.092173    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\" (UniqueName: \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\") pod \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") "
Oct 30 19:47:27.092456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.092442    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\") pod \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") "
Oct 30 19:47:27.092585 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.092550    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\") pod \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\" (UID: \"c9f7532f-d80e-4ed2-a01a-1cfb804460fe\") "
Oct 30 19:47:27.092885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.092861    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca" pod "c9f7532f-d80e-4ed2-a01a-1cfb804460fe" (UID: "c9f7532f-d80e-4ed2-a01a-1cfb804460fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:27.093300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.093276    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:27.095050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.095015    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca" pod "c9f7532f-d80e-4ed2-a01a-1cfb804460fe" (UID: "c9f7532f-d80e-4ed2-a01a-1cfb804460fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:27.095291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.095265    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt" pod "c9f7532f-d80e-4ed2-a01a-1cfb804460fe" (UID: "c9f7532f-d80e-4ed2-a01a-1cfb804460fe"). InnerVolumeSpecName "kube-api-access-c2wzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:27.194195 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.194165    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:27.194312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.194197    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wzt\" (UniqueName: \"kubernetes.io/projected/c9f7532f-d80e-4ed2-a01a-1cfb804460fe-kube-api-access-c2wzt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:27.807826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.807794    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8" event={"ID":"c9f7532f-d80e-4ed2-a01a-1cfb804460fe","Type":"ContainerDied","Data":"9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292"}
Oct 30 19:47:27.807826 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.807815    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"
Oct 30 19:47:27.808030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:27.807840    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d910fcbd34e55810ef078b138738448cff041f65a15b03a88a638e1a4ba4292"
Oct 30 19:47:28.734807 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:47:28.734768    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:47:30.505645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:30.505574    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:47:30.506620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:30.506540    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy" containerID="cri-o://4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04" gracePeriod=30
Oct 30 19:47:35.824147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.824115    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:47:35.878684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878652    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.878830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878776    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.878830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878816    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.878940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878846    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.878940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878901    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.878940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.878924    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.879101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.879003    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.879192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.879175    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.879263 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.879221    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\") pod \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\" (UID: \"3fff4667-95f1-4888-b1ae-fd0c4ba551ac\") "
Oct 30 19:47:35.880128 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.880100    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:35.880442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.880422    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:35.880845 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.880816    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:35.881779 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.881749    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:35.882076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.882046    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:35.883068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.883038    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "kube-api-access-ft6rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:35.883068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.883056    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:35.883544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.883486    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:35.884370 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.884342    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy" pod "3fff4667-95f1-4888-b1ae-fd0c4ba551ac" (UID: "3fff4667-95f1-4888-b1ae-fd0c4ba551ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:35.901543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.901275    4320 generic.go:356] "Generic (PLEG): container finished" podID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerID="4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04" exitCode=0
Oct 30 19:47:35.901543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.901453    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" event={"ID":"3fff4667-95f1-4888-b1ae-fd0c4ba551ac","Type":"ContainerDied","Data":"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"}
Oct 30 19:47:35.901543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.901505    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt" event={"ID":"3fff4667-95f1-4888-b1ae-fd0c4ba551ac","Type":"ContainerDied","Data":"e659f49a41e53a376e2013d933afcb63fed8e836e17c53a38a2b240ca3d03846"}
Oct 30 19:47:35.901543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.901536    4320 scope.go:122] "RemoveContainer" containerID="4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"
Oct 30 19:47:35.903693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.901730    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"
Oct 30 19:47:35.924684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.924667    4320 scope.go:122] "RemoveContainer" containerID="4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"
Oct 30 19:47:35.924985 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:47:35.924963    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04\": container with ID starting with 4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04 not found: ID does not exist" containerID="4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"
Oct 30 19:47:35.925064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.924996    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04"} err="failed to get container status \"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04\": rpc error: code = NotFound desc = could not find container \"4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04\": container with ID starting with 4d6d19a58c5887de282e91d94c0516fb07dd3b9639cf1f5549989b2767f66f04 not found: ID does not exist"
Oct 30 19:47:35.957853 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.957816    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:47:35.961900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.961876    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-bqhrt"]
Oct 30 19:47:35.981357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981331    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981353    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981364    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6rd\" (UniqueName: \"kubernetes.io/projected/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-kube-api-access-ft6rd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981373    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981382    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981389    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981396    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981405    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:35.981519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:35.981413    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fff4667-95f1-4888-b1ae-fd0c4ba551ac-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:36.097969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:36.097887    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" path="/var/lib/kubelet/pods/3fff4667-95f1-4888-b1ae-fd0c4ba551ac/volumes"
Oct 30 19:47:41.013254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.013216    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"]
Oct 30 19:47:41.014111 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014088    4320 cpu_manager.go:414] "containerMap: removing container" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy"
Oct 30 19:47:41.014230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014111    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy"
Oct 30 19:47:41.014230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014177    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c9f7532f-d80e-4ed2-a01a-1cfb804460fe" containerName="revoke-keys"
Oct 30 19:47:41.014230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014186    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f7532f-d80e-4ed2-a01a-1cfb804460fe" containerName="revoke-keys"
Oct 30 19:47:41.014428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014364    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f7532f-d80e-4ed2-a01a-1cfb804460fe" containerName="revoke-keys"
Oct 30 19:47:41.014428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.014385    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fff4667-95f1-4888-b1ae-fd0c4ba551ac" containerName="istio-proxy"
Oct 30 19:47:41.018523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.018503    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.036448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.036412    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"]
Oct 30 19:47:41.042911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.042885    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:41.043038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.042956    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:47:41.057018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.056983    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.057152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.057062    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng584\" (UniqueName: \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.057249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.057148    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.158772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.158743    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.158908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.158794    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng584\" (UniqueName: \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.158908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.158855    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.159380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.159358    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.161342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.161324    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.177759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.177739    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng584\" (UniqueName: \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\") pod \"maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.332797 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.332731    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:41.636161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.636133    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"]
Oct 30 19:47:41.636964 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:47:41.636937    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31161fe8_5ef4_41ad_86b4_674e914dcaf7.slice/crio-aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d WatchSource:0}: Error finding container aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d: Status 404 returned error can't find the container with id aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d
Oct 30 19:47:41.960816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:41.960780    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz" event={"ID":"31161fe8-5ef4-41ad-86b4-674e914dcaf7","Type":"ContainerStarted","Data":"aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d"}
Oct 30 19:47:42.970016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:42.969983    4320 generic.go:356] "Generic (PLEG): container finished" podID="31161fe8-5ef4-41ad-86b4-674e914dcaf7" containerID="eb6063d12547ea9794fb93f39c8c295e34f17d394522422b1673a323fc203620" exitCode=0
Oct 30 19:47:42.970405 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:42.970070    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz" event={"ID":"31161fe8-5ef4-41ad-86b4-674e914dcaf7","Type":"ContainerDied","Data":"eb6063d12547ea9794fb93f39c8c295e34f17d394522422b1673a323fc203620"}
Oct 30 19:47:44.154909 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.154886    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:44.314417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.304022    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\") pod \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") "
Oct 30 19:47:44.314417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.304117    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\" (UniqueName: \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\") pod \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") "
Oct 30 19:47:44.314417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.304356    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\") pod \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\" (UID: \"31161fe8-5ef4-41ad-86b4-674e914dcaf7\") "
Oct 30 19:47:44.314417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.305497    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca" pod "31161fe8-5ef4-41ad-86b4-674e914dcaf7" (UID: "31161fe8-5ef4-41ad-86b4-674e914dcaf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:44.340939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.340816    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca" pod "31161fe8-5ef4-41ad-86b4-674e914dcaf7" (UID: "31161fe8-5ef4-41ad-86b4-674e914dcaf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:44.341121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.340976    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584" pod "31161fe8-5ef4-41ad-86b4-674e914dcaf7" (UID: "31161fe8-5ef4-41ad-86b4-674e914dcaf7"). InnerVolumeSpecName "kube-api-access-ng584". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:44.406566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.406530    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:44.406566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.406561    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng584\" (UniqueName: \"kubernetes.io/projected/31161fe8-5ef4-41ad-86b4-674e914dcaf7-kube-api-access-ng584\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:44.406566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.406571    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31161fe8-5ef4-41ad-86b4-674e914dcaf7-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:44.989770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.989737    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"
Oct 30 19:47:44.989770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.989750    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz" event={"ID":"31161fe8-5ef4-41ad-86b4-674e914dcaf7","Type":"ContainerDied","Data":"aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d"}
Oct 30 19:47:44.989994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:44.989836    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaba32dab16298f4d878c438f7b3f88ae1ffc19dd706e1f32b68f5a4adaa398d"
Oct 30 19:47:46.832449 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:46.832419    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:47:51.514843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:51.514812    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:47:53.329455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.329419    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:47:53.330035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.329692    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" podUID="99ba2e88-aef6-4a17-8489-0b409eceab70" containerName="maas-api" containerID="cri-o://8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854" gracePeriod=30
Oct 30 19:47:53.836623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.835409    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:47:53.923620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.920521    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\") pod \"99ba2e88-aef6-4a17-8489-0b409eceab70\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") "
Oct 30 19:47:53.923620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.920575    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\" (UniqueName: \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\") pod \"99ba2e88-aef6-4a17-8489-0b409eceab70\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") "
Oct 30 19:47:53.923620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.920644    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\") pod \"99ba2e88-aef6-4a17-8489-0b409eceab70\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") "
Oct 30 19:47:53.923620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.920886    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\") pod \"99ba2e88-aef6-4a17-8489-0b409eceab70\" (UID: \"99ba2e88-aef6-4a17-8489-0b409eceab70\") "
Oct 30 19:47:53.923620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.921885    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca" pod "99ba2e88-aef6-4a17-8489-0b409eceab70" (UID: "99ba2e88-aef6-4a17-8489-0b409eceab70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:53.930057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.927872    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls" pod "99ba2e88-aef6-4a17-8489-0b409eceab70" (UID: "99ba2e88-aef6-4a17-8489-0b409eceab70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:53.950619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.935794    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268" pod "99ba2e88-aef6-4a17-8489-0b409eceab70" (UID: "99ba2e88-aef6-4a17-8489-0b409eceab70"). InnerVolumeSpecName "kube-api-access-hf268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:53.975623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:53.951300    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs" pod "99ba2e88-aef6-4a17-8489-0b409eceab70" (UID: "99ba2e88-aef6-4a17-8489-0b409eceab70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:54.022994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.022899    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:54.022994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.022937    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf268\" (UniqueName: \"kubernetes.io/projected/99ba2e88-aef6-4a17-8489-0b409eceab70-kube-api-access-hf268\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:54.022994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.022953    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba2e88-aef6-4a17-8489-0b409eceab70-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:54.022994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.022968    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba2e88-aef6-4a17-8489-0b409eceab70-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:47:54.062820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.062788    4320 generic.go:356] "Generic (PLEG): container finished" podID="99ba2e88-aef6-4a17-8489-0b409eceab70" containerID="8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854" exitCode=0
Oct 30 19:47:54.062994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.062846    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"
Oct 30 19:47:54.062994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.062864    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" event={"ID":"99ba2e88-aef6-4a17-8489-0b409eceab70","Type":"ContainerDied","Data":"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"}
Oct 30 19:47:54.062994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.062921    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6" event={"ID":"99ba2e88-aef6-4a17-8489-0b409eceab70","Type":"ContainerDied","Data":"bfd6d293a0c34919c121ccd6a799b7a49e6cabba2c8c692b2e03635cdef92cd6"}
Oct 30 19:47:54.062994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.062947    4320 scope.go:122] "RemoveContainer" containerID="8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"
Oct 30 19:47:54.075112 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.075096    4320 scope.go:122] "RemoveContainer" containerID="8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"
Oct 30 19:47:54.075383 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:47:54.075366    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854\": container with ID starting with 8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854 not found: ID does not exist" containerID="8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"
Oct 30 19:47:54.075423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.075391    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854"} err="failed to get container status \"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854\": rpc error: code = NotFound desc = could not find container \"8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854\": container with ID starting with 8a6a983b8a48b31330792d234869df00cacd87986f9716ecdfba7d72e7fff854 not found: ID does not exist"
Oct 30 19:47:54.115232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.115205    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:47:54.128413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.128388    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-614c02-558b8f5bdc-hhpg6"]
Oct 30 19:47:54.205431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.205408    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"]
Oct 30 19:47:54.209721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:54.209699    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-614c02-fhjd8"]
Oct 30 19:47:56.100191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:56.100153    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ba2e88-aef6-4a17-8489-0b409eceab70" path="/var/lib/kubelet/pods/99ba2e88-aef6-4a17-8489-0b409eceab70/volumes"
Oct 30 19:47:56.100687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:56.100670    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f7532f-d80e-4ed2-a01a-1cfb804460fe" path="/var/lib/kubelet/pods/c9f7532f-d80e-4ed2-a01a-1cfb804460fe/volumes"
Oct 30 19:47:56.516066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:56.516017    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing" containerID="cri-o://884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454" gracePeriod=25
Oct 30 19:47:56.537359 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:47:56.537327    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:58.646314 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:47:58.646266    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:48:01.173932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:01.173900    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:48:01.174306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:01.174119    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy" containerID="cri-o://cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f" gracePeriod=30
Oct 30 19:48:06.530126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.530094    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:48:06.536071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.536038    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:06.605087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605053    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605098    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605181    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605208    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605261 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605257    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605307    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605326    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605425    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.605517 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.605482    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\") pod \"cdff107f-517b-4063-8019-4c3d05c65e15\" (UID: \"cdff107f-517b-4063-8019-4c3d05c65e15\") "
Oct 30 19:48:06.606622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.606382    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:06.606825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.606792    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:06.607183 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.607156    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:06.607453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.607429    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:06.608168 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.608143    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:06.610684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.610644    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:06.610884 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.610855    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:06.610966 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.610902    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:06.611033 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.610985    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z" pod "cdff107f-517b-4063-8019-4c3d05c65e15" (UID: "cdff107f-517b-4063-8019-4c3d05c65e15"). InnerVolumeSpecName "kube-api-access-xb58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:06.707670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707644    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdff107f-517b-4063-8019-4c3d05c65e15-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707669    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707678    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707686    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb58z\" (UniqueName: \"kubernetes.io/projected/cdff107f-517b-4063-8019-4c3d05c65e15-kube-api-access-xb58z\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707695    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707703    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707711    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdff107f-517b-4063-8019-4c3d05c65e15-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707733    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:06.707818 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:06.707743    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdff107f-517b-4063-8019-4c3d05c65e15-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:07.169385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.169353    4320 generic.go:356] "Generic (PLEG): container finished" podID="cdff107f-517b-4063-8019-4c3d05c65e15" containerID="cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f" exitCode=0
Oct 30 19:48:07.169549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.169398    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"
Oct 30 19:48:07.169549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.169440    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" event={"ID":"cdff107f-517b-4063-8019-4c3d05c65e15","Type":"ContainerDied","Data":"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"}
Oct 30 19:48:07.169549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.169496    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld" event={"ID":"cdff107f-517b-4063-8019-4c3d05c65e15","Type":"ContainerDied","Data":"9de12f57f7169e11e14452bcc8642c147030147d0762bb784f7661719aa877d9"}
Oct 30 19:48:07.169549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.169526    4320 scope.go:122] "RemoveContainer" containerID="cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"
Oct 30 19:48:07.181335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.181310    4320 scope.go:122] "RemoveContainer" containerID="cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"
Oct 30 19:48:07.181593 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:07.181575    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f\": container with ID starting with cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f not found: ID does not exist" containerID="cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"
Oct 30 19:48:07.181678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.181614    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f"} err="failed to get container status \"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f\": rpc error: code = NotFound desc = could not find container \"cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f\": container with ID starting with cad5bb0389c53f34ae4396a942cec7fcf3c0d9c5225afdbc2e36130efac6337f not found: ID does not exist"
Oct 30 19:48:07.234086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.234063    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:48:07.239588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:07.239566    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-mdsld"]
Oct 30 19:48:08.098791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:08.098761    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" path="/var/lib/kubelet/pods/cdff107f-517b-4063-8019-4c3d05c65e15/volumes"
Oct 30 19:48:11.621326 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621287    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"]
Oct 30 19:48:11.621930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621914    4320 cpu_manager.go:414] "containerMap: removing container" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy"
Oct 30 19:48:11.621975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621931    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy"
Oct 30 19:48:11.621975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621955    4320 cpu_manager.go:414] "containerMap: removing container" podUID="31161fe8-5ef4-41ad-86b4-674e914dcaf7" containerName="revoke-keys"
Oct 30 19:48:11.621975 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621960    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31161fe8-5ef4-41ad-86b4-674e914dcaf7" containerName="revoke-keys"
Oct 30 19:48:11.622068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.621999    4320 cpu_manager.go:414] "containerMap: removing container" podUID="99ba2e88-aef6-4a17-8489-0b409eceab70" containerName="maas-api"
Oct 30 19:48:11.622068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.622005    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ba2e88-aef6-4a17-8489-0b409eceab70" containerName="maas-api"
Oct 30 19:48:11.622130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.622097    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ba2e88-aef6-4a17-8489-0b409eceab70" containerName="maas-api"
Oct 30 19:48:11.622130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.622108    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdff107f-517b-4063-8019-4c3d05c65e15" containerName="istio-proxy"
Oct 30 19:48:11.622130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.622118    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="31161fe8-5ef4-41ad-86b4-674e914dcaf7" containerName="revoke-keys"
Oct 30 19:48:11.625751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.625725    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.643465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.643441    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"]
Oct 30 19:48:11.650147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.650123    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:11.650318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.650155    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:48:11.774803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.774777    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fnp\" (UniqueName: \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.775012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.774992    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.775075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.775050    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.876686 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.876583    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.876815 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.876709    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.876862 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.876813    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fnp\" (UniqueName: \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.877363 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.877339    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.879063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.879040    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.895056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.895033    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fnp\" (UniqueName: \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\") pod \"maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:11.940280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:11.940256    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:12.242203 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:48:12.242163    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1389846e_5069_4cec_bd61_6e0dfe2ce1f7.slice/crio-64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c WatchSource:0}: Error finding container 64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c: Status 404 returned error can't find the container with id 64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c
Oct 30 19:48:12.242656 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:12.242632    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"]
Oct 30 19:48:13.324906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.324866    4320 generic.go:356] "Generic (PLEG): container finished" podID="1389846e-5069-4cec-bd61-6e0dfe2ce1f7" containerID="12cf8212fe679b006cb998b69582bc7f4d92f9984fa7200a3bc3bd4b5e66c83b" exitCode=0
Oct 30 19:48:13.325357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.324935    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn" event={"ID":"1389846e-5069-4cec-bd61-6e0dfe2ce1f7","Type":"ContainerDied","Data":"12cf8212fe679b006cb998b69582bc7f4d92f9984fa7200a3bc3bd4b5e66c83b"}
Oct 30 19:48:13.325357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.325020    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn" event={"ID":"1389846e-5069-4cec-bd61-6e0dfe2ce1f7","Type":"ContainerStarted","Data":"64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c"}
Oct 30 19:48:13.604657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.604574    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"]
Oct 30 19:48:13.608175 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.608152    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09feeb-pvhlz"]
Oct 30 19:48:13.769993 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:13.769960    4320 scope.go:122] "RemoveContainer" containerID="bb61e0c6e42d79c0ddb3885332399041c661d1887ab0ab113321cc98f0ae584d"
Oct 30 19:48:14.098144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.098107    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31161fe8-5ef4-41ad-86b4-674e914dcaf7" path="/var/lib/kubelet/pods/31161fe8-5ef4-41ad-86b4-674e914dcaf7/volumes"
Oct 30 19:48:14.502787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.502760    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:14.528014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.527980    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\" (UniqueName: \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\") pod \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") "
Oct 30 19:48:14.528178 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.528147    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\") pod \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") "
Oct 30 19:48:14.528220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.528178    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\") pod \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\" (UID: \"1389846e-5069-4cec-bd61-6e0dfe2ce1f7\") "
Oct 30 19:48:14.528536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.528504    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca" pod "1389846e-5069-4cec-bd61-6e0dfe2ce1f7" (UID: "1389846e-5069-4cec-bd61-6e0dfe2ce1f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:14.530096 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.530058    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp" pod "1389846e-5069-4cec-bd61-6e0dfe2ce1f7" (UID: "1389846e-5069-4cec-bd61-6e0dfe2ce1f7"). InnerVolumeSpecName "kube-api-access-q2fnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:14.530214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.530135    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca" pod "1389846e-5069-4cec-bd61-6e0dfe2ce1f7" (UID: "1389846e-5069-4cec-bd61-6e0dfe2ce1f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:14.630396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.630363    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:14.630396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.630392    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:14.630396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:14.630402    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fnp\" (UniqueName: \"kubernetes.io/projected/1389846e-5069-4cec-bd61-6e0dfe2ce1f7-kube-api-access-q2fnp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:15.342850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:15.342813    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"
Oct 30 19:48:15.342850 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:15.342827    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn" event={"ID":"1389846e-5069-4cec-bd61-6e0dfe2ce1f7","Type":"ContainerDied","Data":"64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c"}
Oct 30 19:48:15.343102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:15.342872    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64803951b7604b3b137b2c0aa52fc98d6bf4bd175ecacb37e7af2209193acd7c"
Oct 30 19:48:16.536130 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:48:16.536021    4320 logging.go:55] [core] [Channel #3173 SubChannel #3174]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.137:50052", ServerName: "10.133.0.137:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.137:50052: connect: connection refused"
Oct 30 19:48:16.800457 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.800378    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:48:16.859529 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.859491    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\" (UniqueName: \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\") pod \"9800c198-34fd-4852-b7ab-450765f5a477\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") "
Oct 30 19:48:16.859754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.859622    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\") pod \"9800c198-34fd-4852-b7ab-450765f5a477\" (UID: \"9800c198-34fd-4852-b7ab-450765f5a477\") "
Oct 30 19:48:16.860020 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.859993    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume" pod "9800c198-34fd-4852-b7ab-450765f5a477" (UID: "9800c198-34fd-4852-b7ab-450765f5a477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:16.860446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.860423    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9800c198-34fd-4852-b7ab-450765f5a477-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:16.861879 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.861849    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t" pod "9800c198-34fd-4852-b7ab-450765f5a477" (UID: "9800c198-34fd-4852-b7ab-450765f5a477"). InnerVolumeSpecName "kube-api-access-rqr6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:16.962253 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:16.962223    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqr6t\" (UniqueName: \"kubernetes.io/projected/9800c198-34fd-4852-b7ab-450765f5a477-kube-api-access-rqr6t\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:17.358294 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.358259    4320 generic.go:356] "Generic (PLEG): container finished" podID="9800c198-34fd-4852-b7ab-450765f5a477" containerID="884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454" exitCode=0
Oct 30 19:48:17.358493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.358341    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"
Oct 30 19:48:17.358493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.358340    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" event={"ID":"9800c198-34fd-4852-b7ab-450765f5a477","Type":"ContainerDied","Data":"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"}
Oct 30 19:48:17.358493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.358477    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" event={"ID":"9800c198-34fd-4852-b7ab-450765f5a477","Type":"ContainerDied","Data":"7145103cbc760ac3cc4aa0e2c54688f83df604b0b531ef915600e2524b9558a4"}
Oct 30 19:48:17.358705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.358521    4320 scope.go:122] "RemoveContainer" containerID="884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"
Oct 30 19:48:17.371277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.371256    4320 scope.go:122] "RemoveContainer" containerID="884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"
Oct 30 19:48:17.371558 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:17.371538    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454\": container with ID starting with 884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454 not found: ID does not exist" containerID="884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"
Oct 30 19:48:17.371679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.371567    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454"} err="failed to get container status \"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454\": rpc error: code = NotFound desc = could not find container \"884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454\": container with ID starting with 884b75bd05d6ce9c261477a1048eb9f90454bb33ab56e8b0ba6dda112fc33454 not found: ID does not exist"
Oct 30 19:48:17.419828 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.419798    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:48:17.424048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.424023    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd"]
Oct 30 19:48:17.536102 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:17.536059    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-09feeb-6d654c9df6-vxkcd" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.137:50052\" within 1s: context deadline exceeded"
Oct 30 19:48:18.097631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:18.097566    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9800c198-34fd-4852-b7ab-450765f5a477" path="/var/lib/kubelet/pods/9800c198-34fd-4852-b7ab-450765f5a477/volumes"
Oct 30 19:48:20.866972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:20.866932    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:48:20.867685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:20.867655    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" containerID="cri-o://217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb" gracePeriod=30
Oct 30 19:48:23.918611 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.918572    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:48:23.919341 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919321    4320 cpu_manager.go:414] "containerMap: removing container" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing"
Oct 30 19:48:23.919521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919501    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing"
Oct 30 19:48:23.919660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919557    4320 cpu_manager.go:414] "containerMap: removing container" podUID="1389846e-5069-4cec-bd61-6e0dfe2ce1f7" containerName="revoke-keys"
Oct 30 19:48:23.919660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919563    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1389846e-5069-4cec-bd61-6e0dfe2ce1f7" containerName="revoke-keys"
Oct 30 19:48:23.919740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919710    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="1389846e-5069-4cec-bd61-6e0dfe2ce1f7" containerName="revoke-keys"
Oct 30 19:48:23.919740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.919728    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="9800c198-34fd-4852-b7ab-450765f5a477" containerName="payload-processing"
Oct 30 19:48:23.924789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.924749    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.939431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.939409    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-f83282-openshift-default-dockercfg-4lzlr\""
Oct 30 19:48:23.955804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.955776    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:48:23.970316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.970287    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.970544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.970514    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.970958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.970936    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cbj\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.971132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.971111    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.971286 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.971269    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.971485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.971466    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.971649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.971630    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.971867 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.971850    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:23.972030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:23.972013    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.073718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.073681    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.073902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.073741    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.073902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.073776    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.073902 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.073808    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074038    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cbj\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074088 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074076    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074110    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074119    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074157    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074242    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074275    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074410    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.074481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.074430    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.075139 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.075104    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.076759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.076727    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.076950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.076929    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.097225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.097150    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cbj\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.097225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.097160    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.241536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.241443    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:24.669454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.669416    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:48:24.670920 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:48:24.670890    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390c2a2d_bc49_4e8e_b78a_52d625e9189b.slice/crio-cd56edfab23083ae437f19fa9c891750e7208e7b52edabd830076c78978c73ed WatchSource:0}: Error finding container cd56edfab23083ae437f19fa9c891750e7208e7b52edabd830076c78978c73ed: Status 404 returned error can't find the container with id cd56edfab23083ae437f19fa9c891750e7208e7b52edabd830076c78978c73ed
Oct 30 19:48:24.674386 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.674348    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:24.674472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.674431    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:24.674511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.674476    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:48:24.934618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.934030    4320 patch_prober.go:29] interesting pod/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:24.934618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:24.934113    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:25.489933 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.489894    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" event={"ID":"390c2a2d-bc49-4e8e-b78a-52d625e9189b","Type":"ContainerStarted","Data":"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"}
Oct 30 19:48:25.490117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.489957    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" event={"ID":"390c2a2d-bc49-4e8e-b78a-52d625e9189b","Type":"ContainerStarted","Data":"cd56edfab23083ae437f19fa9c891750e7208e7b52edabd830076c78978c73ed"}
Oct 30 19:48:25.552579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.552419    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podStartSLOduration=2.55240073 podStartE2EDuration="2.55240073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:25.550050132 +0000 UTC m=+2653.585562597" watchObservedRunningTime="2026-10-30 19:48:25.55240073 +0000 UTC m=+2653.587913196"
Oct 30 19:48:25.707562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.707524    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"]
Oct 30 19:48:25.712295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.712265    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.729448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.729188    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:48:25.729448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.729252    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:25.731568 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.731543    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"]
Oct 30 19:48:25.819693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.817509    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.819693 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.817657    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzt6\" (UniqueName: \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.821117 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.821073    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.922470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.922439    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.922621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.922571    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzt6\" (UniqueName: \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.922725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.922703    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.923212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.923183    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.925939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.925910    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:25.941543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:25.941521    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzt6\" (UniqueName: \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\") pod \"maas-api-revoke-sub-3a6aad54-xsb2z\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:26.028022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.027988    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:26.210389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.206909    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:48:26.242417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.242386    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:26.243646 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.243618    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:26.243885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.243855    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:26.262995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.262968    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.263142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263113    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.263314 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263285    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.263526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263506    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:26.263623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263582    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.263773 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263757    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263840    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.264144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263921    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263960    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.263997    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264144 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.264120    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264465 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.264153    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\") pod \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\" (UID: \"ab5e966a-d345-47d6-bbd2-997b7fe2d83b\") "
Oct 30 19:48:26.264817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.264796    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.265126 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.265102    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.265252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.265236    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.265354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.265339    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.265464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.265451    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.265844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.265743    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.268543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.268517    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:26.290995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.289663    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:26.290995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.290764    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "kube-api-access-zhnh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:26.306789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.306755    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token" pod "ab5e966a-d345-47d6-bbd2-997b7fe2d83b" (UID: "ab5e966a-d345-47d6-bbd2-997b7fe2d83b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:26.367667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.367579    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.367667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.367643    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.367667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.367663    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.367858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.367673    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.367858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.367682    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhnh7\" (UniqueName: \"kubernetes.io/projected/ab5e966a-d345-47d6-bbd2-997b7fe2d83b-kube-api-access-zhnh7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:26.411744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.411710    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"]
Oct 30 19:48:26.413135 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:48:26.413104    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7381152e_2c68_41ef_9cbb_5fccf02aa6f1.slice/crio-758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4 WatchSource:0}: Error finding container 758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4: Status 404 returned error can't find the container with id 758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4
Oct 30 19:48:26.509108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.509063    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z" event={"ID":"7381152e-2c68-41ef-9cbb-5fccf02aa6f1","Type":"ContainerStarted","Data":"758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4"}
Oct 30 19:48:26.511055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.511025    4320 generic.go:356] "Generic (PLEG): container finished" podID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerID="217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb" exitCode=0
Oct 30 19:48:26.511213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.511081    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"
Oct 30 19:48:26.511213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.511081    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerDied","Data":"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"}
Oct 30 19:48:26.511213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.511206    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m" event={"ID":"ab5e966a-d345-47d6-bbd2-997b7fe2d83b","Type":"ContainerDied","Data":"76c072a10e050ff460c93f78fff9530edef85c62719e0a986dc1119654784a3c"}
Oct 30 19:48:26.511358 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.511229    4320 scope.go:122] "RemoveContainer" containerID="217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"
Oct 30 19:48:26.523508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.523493    4320 scope.go:122] "RemoveContainer" containerID="0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"
Oct 30 19:48:26.532959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.532942    4320 scope.go:122] "RemoveContainer" containerID="217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"
Oct 30 19:48:26.533233 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:26.533210    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb\": container with ID starting with 217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb not found: ID does not exist" containerID="217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"
Oct 30 19:48:26.533269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.533243    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb"} err="failed to get container status \"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb\": rpc error: code = NotFound desc = could not find container \"217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb\": container with ID starting with 217bab76f916205c02cc475cb5f7abeabc8fd54a70960274047125c8fc58cacb not found: ID does not exist"
Oct 30 19:48:26.533269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.533259    4320 scope.go:122] "RemoveContainer" containerID="0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"
Oct 30 19:48:26.533508 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:26.533490    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355\": container with ID starting with 0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355 not found: ID does not exist" containerID="0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"
Oct 30 19:48:26.533543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.533517    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355"} err="failed to get container status \"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355\": rpc error: code = NotFound desc = could not find container \"0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355\": container with ID starting with 0bd53b02729e4e9f65abab1908b72637f944b7876e02df0cf7dfa56abd7c5355 not found: ID does not exist"
Oct 30 19:48:26.568151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.568128    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:48:26.600589 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:26.600554    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-09feeb-openshift-default-77dd7c9c95-r878m"]
Oct 30 19:48:27.242481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:27.242444    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:27.242918 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:27.242504    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:27.521551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:27.521519    4320 generic.go:356] "Generic (PLEG): container finished" podID="7381152e-2c68-41ef-9cbb-5fccf02aa6f1" containerID="a401ae3b55bdf8a9d9c29caa30c6dd6070443e30f15ba19641fee8cebd9a07a2" exitCode=0
Oct 30 19:48:27.521707 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:27.521628    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z" event={"ID":"7381152e-2c68-41ef-9cbb-5fccf02aa6f1","Type":"ContainerDied","Data":"a401ae3b55bdf8a9d9c29caa30c6dd6070443e30f15ba19641fee8cebd9a07a2"}
Oct 30 19:48:28.268627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.259537    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:28.268627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.259612    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:28.318636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.316213    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" path="/var/lib/kubelet/pods/ab5e966a-d345-47d6-bbd2-997b7fe2d83b/volumes"
Oct 30 19:48:28.652081 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:28.650878    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:48:28.730162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.730136    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:28.872669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.868562    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\") pod \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") "
Oct 30 19:48:28.873050 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.873020    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\") pod \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") "
Oct 30 19:48:28.889655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.883944    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca" pod "7381152e-2c68-41ef-9cbb-5fccf02aa6f1" (UID: "7381152e-2c68-41ef-9cbb-5fccf02aa6f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:28.899717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.899669    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca" pod "7381152e-2c68-41ef-9cbb-5fccf02aa6f1" (UID: "7381152e-2c68-41ef-9cbb-5fccf02aa6f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:28.899859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.899834    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\" (UniqueName: \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\") pod \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\" (UID: \"7381152e-2c68-41ef-9cbb-5fccf02aa6f1\") "
Oct 30 19:48:28.900688 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.900663    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:28.900807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.900691    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:28.901930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:28.901907    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6" pod "7381152e-2c68-41ef-9cbb-5fccf02aa6f1" (UID: "7381152e-2c68-41ef-9cbb-5fccf02aa6f1"). InnerVolumeSpecName "kube-api-access-pfzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:29.002473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.002439    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfzt6\" (UniqueName: \"kubernetes.io/projected/7381152e-2c68-41ef-9cbb-5fccf02aa6f1-kube-api-access-pfzt6\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:29.242280 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.242240    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:29.242472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.242294    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:29.539329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.539260    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"
Oct 30 19:48:29.539329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.539277    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z" event={"ID":"7381152e-2c68-41ef-9cbb-5fccf02aa6f1","Type":"ContainerDied","Data":"758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4"}
Oct 30 19:48:29.539329 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:29.539321    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758b225f42250d36f15adde6a25e15efb421d1f34e77335253e78fcece6802d4"
Oct 30 19:48:30.241780 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:30.241741    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:30.241976 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:30.241801    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:31.242038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:31.242003    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:48:31.242431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:31.242065    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 19:48:32.247729 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:32.247696    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:32.248148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:32.248092    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:32.248812 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:32.248784    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:48:46.030715 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:46.030680    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"]
Oct 30 19:48:46.034368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:46.034342    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3504b-v8pnn"]
Oct 30 19:48:46.098395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:46.098358    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1389846e-5069-4cec-bd61-6e0dfe2ce1f7" path="/var/lib/kubelet/pods/1389846e-5069-4cec-bd61-6e0dfe2ce1f7/volumes"
Oct 30 19:48:52.123189 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.123155    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:48:52.306618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.305828    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"]
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.312900    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.312926    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.312986    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7381152e-2c68-41ef-9cbb-5fccf02aa6f1" containerName="revoke-keys"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.312994    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7381152e-2c68-41ef-9cbb-5fccf02aa6f1" containerName="revoke-keys"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.313028    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.313035    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.313244    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.313260    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5e966a-d345-47d6-bbd2-997b7fe2d83b" containerName="istio-proxy"
Oct 30 19:48:52.317276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.313279    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7381152e-2c68-41ef-9cbb-5fccf02aa6f1" containerName="revoke-keys"
Oct 30 19:48:52.325833 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.325809    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.327182 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.327151    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"]
Oct 30 19:48:52.342722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.342019    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:52.342722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.342096    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:48:52.450277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.450247    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.450461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.450411    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn4c\" (UniqueName: \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.450461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.450459    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.552058 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.552021    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.552262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.552190    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn4c\" (UniqueName: \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.552262 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.552256    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.553122 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.553092    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.554460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.554429    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.571414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.571388    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn4c\" (UniqueName: \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\") pod \"maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.647343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.647318    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:52.958849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:52.958822    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"]
Oct 30 19:48:52.959662 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:48:52.959632    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6d9bef_3acd_4a10_8f3f_a18ea0a17bbd.slice/crio-62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f WatchSource:0}: Error finding container 62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f: Status 404 returned error can't find the container with id 62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f
Oct 30 19:48:53.751690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:53.751651    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr" event={"ID":"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd","Type":"ContainerStarted","Data":"439a1dd00837c45e9aa981a57236e7c3e69f84f8e6175d275c8493624cd7f4b3"}
Oct 30 19:48:53.752084 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:53.751718    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr" event={"ID":"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd","Type":"ContainerStarted","Data":"62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f"}
Oct 30 19:48:53.794911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:53.794864    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr" podStartSLOduration=1.7948514709999999 podStartE2EDuration="1.794851471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:53.789978544 +0000 UTC m=+2681.825491009" watchObservedRunningTime="2026-10-30 19:48:53.794851471 +0000 UTC m=+2681.830363935"
Oct 30 19:48:54.760731 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:54.760695    4320 generic.go:356] "Generic (PLEG): container finished" podID="7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" containerID="439a1dd00837c45e9aa981a57236e7c3e69f84f8e6175d275c8493624cd7f4b3" exitCode=0
Oct 30 19:48:54.761188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:54.760749    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr" event={"ID":"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd","Type":"ContainerDied","Data":"439a1dd00837c45e9aa981a57236e7c3e69f84f8e6175d275c8493624cd7f4b3"}
Oct 30 19:48:56.069090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.069064    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:56.099312 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.099286    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\" (UniqueName: \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\") pod \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") "
Oct 30 19:48:56.099432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.099384    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\") pod \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") "
Oct 30 19:48:56.099432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.099421    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\") pod \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\" (UID: \"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd\") "
Oct 30 19:48:56.099842 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.099818    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca" pod "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" (UID: "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:56.100026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.100009    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:56.101375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.101351    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c" pod "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" (UID: "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd"). InnerVolumeSpecName "kube-api-access-jkn4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:56.101454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.101410    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca" pod "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" (UID: "7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:56.201324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.201292    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkn4c\" (UniqueName: \"kubernetes.io/projected/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-kube-api-access-jkn4c\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:56.201324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.201328    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:48:56.778664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.778626    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr" event={"ID":"7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd","Type":"ContainerDied","Data":"62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f"}
Oct 30 19:48:56.778664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.778663    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"
Oct 30 19:48:56.778978 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:48:56.778679    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b06ee0003b0cad1961892b177660fc84a7989c5bea03ea0cfa962a8575958f"
Oct 30 19:48:58.644568 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:48:58.644529    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:49:06.114803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:06.114771    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"]
Oct 30 19:49:06.126740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:06.126715    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f6cef-8tlzr"]
Oct 30 19:49:07.153475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.153414    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" podUID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" containerName="main" containerID="cri-o://45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c" gracePeriod=45
Oct 30 19:49:07.445302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.445274    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:49:07.509065 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509030    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509090    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\" (UniqueName: \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509169    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509269    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509357 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509328    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509405    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\") pod \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\" (UID: \"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc\") "
Oct 30 19:49:07.509723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509696    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.509945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.509904    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.510133 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.510110    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.510224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.510141    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.510353 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.510334    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.512053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.512030    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.512208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.512173    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "kube-api-access-v9lmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.512653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.512592    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs" pod "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" (UID: "b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:07.611542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.611507    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.611542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.611534    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.611542 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.611547    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lmq\" (UniqueName: \"kubernetes.io/projected/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-kube-api-access-v9lmq\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.611781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.611556    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:07.919919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.919890    4320 generic.go:356] "Generic (PLEG): container finished" podID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" containerID="45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c" exitCode=0
Oct 30 19:49:07.920063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.919933    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"
Oct 30 19:49:07.920063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.919975    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" event={"ID":"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc","Type":"ContainerDied","Data":"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"}
Oct 30 19:49:07.920063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.920018    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm" event={"ID":"b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc","Type":"ContainerDied","Data":"3c00bef936c8792f01eff4c5a2ba08c61349344b9d0c378b1f7b0a07aeb8a484"}
Oct 30 19:49:07.920063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.920034    4320 scope.go:122] "RemoveContainer" containerID="45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"
Oct 30 19:49:07.931472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.931454    4320 scope.go:122] "RemoveContainer" containerID="45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"
Oct 30 19:49:07.931745 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:49:07.931725    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c\": container with ID starting with 45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c not found: ID does not exist" containerID="45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"
Oct 30 19:49:07.931809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.931751    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c"} err="failed to get container status \"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c\": rpc error: code = NotFound desc = could not find container \"45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c\": container with ID starting with 45569b312ac57af0dd80ef6ab948d837682b85c21ec5a5e6f1cde113d101997c not found: ID does not exist"
Oct 30 19:49:07.972388 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.972364    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:49:07.978198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:07.978178    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-576a62/test-model-576a62-kserve-56565fd85c-9vlcm"]
Oct 30 19:49:08.109590 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:08.109543    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" path="/var/lib/kubelet/pods/7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd/volumes"
Oct 30 19:49:08.110265 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:08.110237    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" path="/var/lib/kubelet/pods/b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc/volumes"
Oct 30 19:49:22.028635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:22.028581    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"]
Oct 30 19:49:22.040409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:22.040375    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce6504-9c5ws"]
Oct 30 19:49:22.099475 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:22.099448    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915" path="/var/lib/kubelet/pods/b75c2f6c-e0ee-4a11-b1e9-7bb58fa33915/volumes"
Oct 30 19:49:23.010587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.010551    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:49:23.579260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.579224    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"]
Oct 30 19:49:23.580000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.579977    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" containerName="main"
Oct 30 19:49:23.580000 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.580000    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" containerName="main"
Oct 30 19:49:23.580149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.580038    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" containerName="revoke-keys"
Oct 30 19:49:23.580149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.580044    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" containerName="revoke-keys"
Oct 30 19:49:23.580149 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.580139    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f3e86b-6f24-4e7f-8bb9-88a4f36a32dc" containerName="main"
Oct 30 19:49:23.580299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.580152    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6d9bef-3acd-4a10-8f3f-a18ea0a17bbd" containerName="revoke-keys"
Oct 30 19:49:23.585872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.585850    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.602309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.602284    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"]
Oct 30 19:49:23.607644 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.607583    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvj2\" (UniqueName: \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.607875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.607849    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.607998 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.607977    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.608864 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.608843    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:23.609007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.608878    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:49:23.708658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.708621    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.708787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.708687    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvj2\" (UniqueName: \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.708972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.708944    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.709393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.709371    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.711224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.711202    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.728899 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.728878    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvj2\" (UniqueName: \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\") pod \"maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:23.902447 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:23.902427    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:24.213782 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:24.213757    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"]
Oct 30 19:49:24.215130 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:49:24.215106    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0fa17a_87fb_4263_bfdb_642902a4b57f.slice/crio-70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e WatchSource:0}: Error finding container 70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e: Status 404 returned error can't find the container with id 70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e
Oct 30 19:49:25.134714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:25.134674    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6" event={"ID":"3c0fa17a-87fb-4263-bfdb-642902a4b57f","Type":"ContainerStarted","Data":"8c4816167f578d5d4da637e9844dd02dffdeb0ce92133cdbbd897bffe5231333"}
Oct 30 19:49:25.135544 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:25.135509    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6" event={"ID":"3c0fa17a-87fb-4263-bfdb-642902a4b57f","Type":"ContainerStarted","Data":"70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e"}
Oct 30 19:49:25.189627 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:25.187816    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6" podStartSLOduration=2.187795817 podStartE2EDuration="2.187795817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:25.185011896 +0000 UTC m=+2713.220524360" watchObservedRunningTime="2026-10-30 19:49:25.187795817 +0000 UTC m=+2713.223308282"
Oct 30 19:49:26.144471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:26.144436    4320 generic.go:356] "Generic (PLEG): container finished" podID="3c0fa17a-87fb-4263-bfdb-642902a4b57f" containerID="8c4816167f578d5d4da637e9844dd02dffdeb0ce92133cdbbd897bffe5231333" exitCode=0
Oct 30 19:49:26.144955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:26.144482    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6" event={"ID":"3c0fa17a-87fb-4263-bfdb-642902a4b57f","Type":"ContainerDied","Data":"8c4816167f578d5d4da637e9844dd02dffdeb0ce92133cdbbd897bffe5231333"}
Oct 30 19:49:27.326289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.326261    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:27.381187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.381154    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\") pod \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") "
Oct 30 19:49:27.381374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.381213    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\" (UniqueName: \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\") pod \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") "
Oct 30 19:49:27.381456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.381434    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\") pod \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\" (UID: \"3c0fa17a-87fb-4263-bfdb-642902a4b57f\") "
Oct 30 19:49:27.381926 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.381898    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca" pod "3c0fa17a-87fb-4263-bfdb-642902a4b57f" (UID: "3c0fa17a-87fb-4263-bfdb-642902a4b57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:27.384085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.384057    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2" pod "3c0fa17a-87fb-4263-bfdb-642902a4b57f" (UID: "3c0fa17a-87fb-4263-bfdb-642902a4b57f"). InnerVolumeSpecName "kube-api-access-jcvj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.384228 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.384206    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca" pod "3c0fa17a-87fb-4263-bfdb-642902a4b57f" (UID: "3c0fa17a-87fb-4263-bfdb-642902a4b57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:27.483657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.483560    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:27.483657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.483585    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvj2\" (UniqueName: \"kubernetes.io/projected/3c0fa17a-87fb-4263-bfdb-642902a4b57f-kube-api-access-jcvj2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:27.483657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:27.483627    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0fa17a-87fb-4263-bfdb-642902a4b57f-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:28.163803 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:28.163770    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6" event={"ID":"3c0fa17a-87fb-4263-bfdb-642902a4b57f","Type":"ContainerDied","Data":"70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e"}
Oct 30 19:49:28.163951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:28.163832    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70cc2107f9b5755d95d978e9a14a7cc02f409e0bb4febeace0921ff68f4d983e"
Oct 30 19:49:28.163951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:28.163778    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"
Oct 30 19:49:28.644129 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:49:28.644092    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:49:30.354957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.354926    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"]
Oct 30 19:49:30.355563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.355549    4320 cpu_manager.go:414] "containerMap: removing container" podUID="3c0fa17a-87fb-4263-bfdb-642902a4b57f" containerName="revoke-keys"
Oct 30 19:49:30.355664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.355564    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0fa17a-87fb-4263-bfdb-642902a4b57f" containerName="revoke-keys"
Oct 30 19:49:30.355708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.355697    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0fa17a-87fb-4263-bfdb-642902a4b57f" containerName="revoke-keys"
Oct 30 19:49:30.358771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.358756    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.378037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.378010    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"]
Oct 30 19:49:30.385684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.385219    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:30.385684 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.385278    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:49:30.420298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.420267    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.420462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.420308    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.420539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.420471    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5sq\" (UniqueName: \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.521290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.521262    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.521422 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.521314    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.521487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.521439    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5sq\" (UniqueName: \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.521962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.521944    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.523553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.523536    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.540302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.540276    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5sq\" (UniqueName: \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\") pod \"maas-api-revoke-sub-8005804d-mrfgp\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.672424 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.672403    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:30.974018 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:30.973985    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"]
Oct 30 19:49:30.974875 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:49:30.974847    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0806de01_f769_4cfc_8f34_2892a9c7219a.slice/crio-8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12 WatchSource:0}: Error finding container 8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12: Status 404 returned error can't find the container with id 8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12
Oct 30 19:49:31.189882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:31.189850    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp" event={"ID":"0806de01-f769-4cfc-8f34-2892a9c7219a","Type":"ContainerStarted","Data":"8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12"}
Oct 30 19:49:32.198650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:32.198581    4320 generic.go:356] "Generic (PLEG): container finished" podID="0806de01-f769-4cfc-8f34-2892a9c7219a" containerID="69ac61a8c0c08abd69473922b506dc7ebefdac730720c8c297725d993e3e71cb" exitCode=0
Oct 30 19:49:32.198650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:32.198637    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp" event={"ID":"0806de01-f769-4cfc-8f34-2892a9c7219a","Type":"ContainerDied","Data":"69ac61a8c0c08abd69473922b506dc7ebefdac730720c8c297725d993e3e71cb"}
Oct 30 19:49:33.028240 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.028199    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"]
Oct 30 19:49:33.039702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.039673    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d177766-tpcp4"]
Oct 30 19:49:33.359264 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.359243    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:33.457012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.456983    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\" (UniqueName: \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\") pod \"0806de01-f769-4cfc-8f34-2892a9c7219a\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") "
Oct 30 19:49:33.457191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.457129    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\") pod \"0806de01-f769-4cfc-8f34-2892a9c7219a\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") "
Oct 30 19:49:33.457272 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.457251    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\") pod \"0806de01-f769-4cfc-8f34-2892a9c7219a\" (UID: \"0806de01-f769-4cfc-8f34-2892a9c7219a\") "
Oct 30 19:49:33.457516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.457479    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca" pod "0806de01-f769-4cfc-8f34-2892a9c7219a" (UID: "0806de01-f769-4cfc-8f34-2892a9c7219a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:33.457908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.457888    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:33.459414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.459388    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq" pod "0806de01-f769-4cfc-8f34-2892a9c7219a" (UID: "0806de01-f769-4cfc-8f34-2892a9c7219a"). InnerVolumeSpecName "kube-api-access-tx5sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:33.459737 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.459710    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca" pod "0806de01-f769-4cfc-8f34-2892a9c7219a" (UID: "0806de01-f769-4cfc-8f34-2892a9c7219a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:33.559769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.559740    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0806de01-f769-4cfc-8f34-2892a9c7219a-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:33.559769 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:33.559772    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx5sq\" (UniqueName: \"kubernetes.io/projected/0806de01-f769-4cfc-8f34-2892a9c7219a-kube-api-access-tx5sq\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:34.097534    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8597e55-1a5a-4322-8d85-916ad4daf301" path="/var/lib/kubelet/pods/a8597e55-1a5a-4322-8d85-916ad4daf301/volumes"
Oct 30 19:49:34.223173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:34.223140    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"
Oct 30 19:49:34.223173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:34.223167    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp" event={"ID":"0806de01-f769-4cfc-8f34-2892a9c7219a","Type":"ContainerDied","Data":"8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12"}
Oct 30 19:49:34.223420 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:34.223223    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8503f101a389573125ff22fbdbba756c4a5bcfe3895762a7081376a203639c12"
Oct 30 19:49:40.774418 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.774384    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"]
Oct 30 19:49:40.775042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.775028    4320 cpu_manager.go:414] "containerMap: removing container" podUID="0806de01-f769-4cfc-8f34-2892a9c7219a" containerName="revoke-keys"
Oct 30 19:49:40.775098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.775044    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0806de01-f769-4cfc-8f34-2892a9c7219a" containerName="revoke-keys"
Oct 30 19:49:40.775169 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.775159    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="0806de01-f769-4cfc-8f34-2892a9c7219a" containerName="revoke-keys"
Oct 30 19:49:40.778945 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.778923    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:40.797306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.797281    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"]
Oct 30 19:49:40.804988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.804967    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:40.805087 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.805001    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:49:40.879669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.866022    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:40.881274 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.881240    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lp2v\" (UniqueName: \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:40.885333 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:40.881781    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.010623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.009764    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.010623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.009926    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lp2v\" (UniqueName: \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.010623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.010014    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.022557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.022526    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.023442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.023414    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.071416 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.071328    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lp2v\" (UniqueName: \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\") pod \"maas-api-revoke-sub-4fada187-4p68z\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.094141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.094101    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:41.395567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:41.395537    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"]
Oct 30 19:49:41.396983 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:49:41.396958    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468bab8e_aad7_4227_b693_9952d9166a50.slice/crio-30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a WatchSource:0}: Error finding container 30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a: Status 404 returned error can't find the container with id 30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a
Oct 30 19:49:42.281053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:42.281020    4320 generic.go:356] "Generic (PLEG): container finished" podID="468bab8e-aad7-4227-b693-9952d9166a50" containerID="993374971a01f2f93284275d0d634a9b96b17b17a0e1cf52f71a3cab28ad4091" exitCode=0
Oct 30 19:49:42.281389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:42.281086    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z" event={"ID":"468bab8e-aad7-4227-b693-9952d9166a50","Type":"ContainerDied","Data":"993374971a01f2f93284275d0d634a9b96b17b17a0e1cf52f71a3cab28ad4091"}
Oct 30 19:49:42.281389 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:42.281137    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z" event={"ID":"468bab8e-aad7-4227-b693-9952d9166a50","Type":"ContainerStarted","Data":"30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a"}
Oct 30 19:49:43.484983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.484959    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:43.543809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.543779    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\") pod \"468bab8e-aad7-4227-b693-9952d9166a50\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") "
Oct 30 19:49:43.543964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.543852    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\") pod \"468bab8e-aad7-4227-b693-9952d9166a50\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") "
Oct 30 19:49:43.543964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.543950    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\" (UniqueName: \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\") pod \"468bab8e-aad7-4227-b693-9952d9166a50\" (UID: \"468bab8e-aad7-4227-b693-9952d9166a50\") "
Oct 30 19:49:43.544188 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.544161    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca" pod "468bab8e-aad7-4227-b693-9952d9166a50" (UID: "468bab8e-aad7-4227-b693-9952d9166a50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:43.544781 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.544714    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:43.546141 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.546108    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca" pod "468bab8e-aad7-4227-b693-9952d9166a50" (UID: "468bab8e-aad7-4227-b693-9952d9166a50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:43.546733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.546703    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v" pod "468bab8e-aad7-4227-b693-9952d9166a50" (UID: "468bab8e-aad7-4227-b693-9952d9166a50"). InnerVolumeSpecName "kube-api-access-9lp2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:43.646461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.646436    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468bab8e-aad7-4227-b693-9952d9166a50-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:43.646461 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:43.646459    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lp2v\" (UniqueName: \"kubernetes.io/projected/468bab8e-aad7-4227-b693-9952d9166a50-kube-api-access-9lp2v\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:49:44.302289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:44.302257    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"
Oct 30 19:49:44.302470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:44.302256    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z" event={"ID":"468bab8e-aad7-4227-b693-9952d9166a50","Type":"ContainerDied","Data":"30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a"}
Oct 30 19:49:44.302470 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:44.302381    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cc3bbc194a76c3eef5290cf01a0c0d5cc267e4fc2d6049727fc46f3032949a"
Oct 30 19:49:57.322411 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.322338    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:49:57.323003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.322985    4320 cpu_manager.go:414] "containerMap: removing container" podUID="468bab8e-aad7-4227-b693-9952d9166a50" containerName="revoke-keys"
Oct 30 19:49:57.323049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.323004    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468bab8e-aad7-4227-b693-9952d9166a50" containerName="revoke-keys"
Oct 30 19:49:57.323125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.323116    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="468bab8e-aad7-4227-b693-9952d9166a50" containerName="revoke-keys"
Oct 30 19:49:57.327051 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.327031    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.344525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.344506    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:49:57.351361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.351335    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-5250ee-serving-cert\""
Oct 30 19:49:57.467148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.467116    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.467305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.467166    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.467305 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.467286    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mn6\" (UniqueName: \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.467409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.467394    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.568986 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.568950    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.569196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.569022    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.569196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.569106    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mn6\" (UniqueName: \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.569323 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.569232    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.570961 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.570920    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.573208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.573132    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.576115 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.576087    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.592234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.591312    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mn6\" (UniqueName: \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\") pod \"maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.641459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.641430    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:57.948045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.948017    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:49:57.948944 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:49:57.948914    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51278123_0a2f_419d_8204_5065b601c10b.slice/crio-eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7 WatchSource:0}: Error finding container eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7: Status 404 returned error can't find the container with id eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7
Oct 30 19:49:57.950396 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:57.950378    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:49:58.453273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:58.453231    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" event={"ID":"51278123-0a2f-419d-8204-5065b601c10b","Type":"ContainerStarted","Data":"4c1c5b165fd4a6fe9d4104724db14c9dd5119d7fd8fdfb927483b6698f31ab77"}
Oct 30 19:49:58.453740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:58.453309    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" event={"ID":"51278123-0a2f-419d-8204-5065b601c10b","Type":"ContainerStarted","Data":"eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7"}
Oct 30 19:49:58.453740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:58.453490    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:49:58.497210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:49:58.497117    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" podStartSLOduration=1.208487227 podStartE2EDuration="1.497103522s" totalImagesPullingTime="288.616295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:58.492793437 +0000 UTC m=+2746.528305902" watchObservedRunningTime="2026-10-30 19:49:58.497103522 +0000 UTC m=+2746.532616017"
Oct 30 19:49:58.645571 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:49:58.645534    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:50:04.467816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:04.467785    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:50:07.220140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:07.220106    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"]
Oct 30 19:50:07.229395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:07.229368    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-576a62-bn9v6"]
Oct 30 19:50:08.099913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:08.099874    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0fa17a-87fb-4263-bfdb-642902a4b57f" path="/var/lib/kubelet/pods/3c0fa17a-87fb-4263-bfdb-642902a4b57f/volumes"
Oct 30 19:50:09.798345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:09.798312    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:50:16.395056 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.395024    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"]
Oct 30 19:50:16.399489 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.399466    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.414038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.414011    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54cx\" (UniqueName: \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.414271 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.414248    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.414423 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.414403    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.451739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.451694    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"]
Oct 30 19:50:16.502722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.502687    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:16.502722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.502713    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:50:16.516522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.516490    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54cx\" (UniqueName: \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.516689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.516552    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.516689 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.516588    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.517336 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.517317    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.519186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.519162    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.535974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.535948    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54cx\" (UniqueName: \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\") pod \"maas-api-revoke-sub-82c2283c-ppjz2\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:16.715723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:16.715653    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:17.019021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:17.018991    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"]
Oct 30 19:50:17.020420 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:50:17.020393    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffac3574_413f_48ec_8df9_7d6723a20d1c.slice/crio-18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a WatchSource:0}: Error finding container 18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a: Status 404 returned error can't find the container with id 18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a
Oct 30 19:50:17.676937 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:17.676904    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2" event={"ID":"ffac3574-413f-48ec-8df9-7d6723a20d1c","Type":"ContainerStarted","Data":"887e95c73e4ba68c8573a128c0c02de9e5d6cc37ab52afd754ab3dcf6faaa8a0"}
Oct 30 19:50:17.677334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:17.676958    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2" event={"ID":"ffac3574-413f-48ec-8df9-7d6723a20d1c","Type":"ContainerStarted","Data":"18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a"}
Oct 30 19:50:18.685187 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:18.685151    4320 generic.go:356] "Generic (PLEG): container finished" podID="ffac3574-413f-48ec-8df9-7d6723a20d1c" containerID="887e95c73e4ba68c8573a128c0c02de9e5d6cc37ab52afd754ab3dcf6faaa8a0" exitCode=0
Oct 30 19:50:18.685565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:18.685240    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2" event={"ID":"ffac3574-413f-48ec-8df9-7d6723a20d1c","Type":"ContainerDied","Data":"887e95c73e4ba68c8573a128c0c02de9e5d6cc37ab52afd754ab3dcf6faaa8a0"}
Oct 30 19:50:19.854135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.854110    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:19.990946 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.990840    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\") pod \"ffac3574-413f-48ec-8df9-7d6723a20d1c\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") "
Oct 30 19:50:19.991196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.990956    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\" (UniqueName: \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\") pod \"ffac3574-413f-48ec-8df9-7d6723a20d1c\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") "
Oct 30 19:50:19.991196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.991150    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\") pod \"ffac3574-413f-48ec-8df9-7d6723a20d1c\" (UID: \"ffac3574-413f-48ec-8df9-7d6723a20d1c\") "
Oct 30 19:50:19.992196 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.992166    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca" pod "ffac3574-413f-48ec-8df9-7d6723a20d1c" (UID: "ffac3574-413f-48ec-8df9-7d6723a20d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:19.994098 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.994065    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca" pod "ffac3574-413f-48ec-8df9-7d6723a20d1c" (UID: "ffac3574-413f-48ec-8df9-7d6723a20d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:19.994558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:19.994528    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx" pod "ffac3574-413f-48ec-8df9-7d6723a20d1c" (UID: "ffac3574-413f-48ec-8df9-7d6723a20d1c"). InnerVolumeSpecName "kube-api-access-z54cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:20.095030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.094952    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:20.095030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.094988    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffac3574-413f-48ec-8df9-7d6723a20d1c-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:20.095030 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.095004    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54cx\" (UniqueName: \"kubernetes.io/projected/ffac3574-413f-48ec-8df9-7d6723a20d1c-kube-api-access-z54cx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:20.700721 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.700695    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"
Oct 30 19:50:20.700875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.700728    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2" event={"ID":"ffac3574-413f-48ec-8df9-7d6723a20d1c","Type":"ContainerDied","Data":"18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a"}
Oct 30 19:50:20.700875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:20.700785    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18eb20cba8e0c7cd7a95e0963ce29b8da2fc96c090e932f5ff1e4820cd4da60a"
Oct 30 19:50:26.690717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.690682    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"]
Oct 30 19:50:26.691302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.691287    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ffac3574-413f-48ec-8df9-7d6723a20d1c" containerName="revoke-keys"
Oct 30 19:50:26.691352 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.691303    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffac3574-413f-48ec-8df9-7d6723a20d1c" containerName="revoke-keys"
Oct 30 19:50:26.691430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.691421    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffac3574-413f-48ec-8df9-7d6723a20d1c" containerName="revoke-keys"
Oct 30 19:50:26.695324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.695307    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.714367 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.714342    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"]
Oct 30 19:50:26.721878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.720421    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:26.721878 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.720549    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:50:26.778994 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.778963    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.779121 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.779106    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.779219 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.779206    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.880844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.880810    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.880965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.880946    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.881080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.881065    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.881455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.881434    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.883545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.883520    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:26.900434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:26.900375    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\") pod \"maas-api-revoke-sub-7f4ac577-fsnn7\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:27.010525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:27.010467    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:27.338145 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:27.338117    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"]
Oct 30 19:50:27.338751 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:50:27.338719    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3043364_7216_4a0d_83d9_ad302371c599.slice/crio-e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25 WatchSource:0}: Error finding container e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25: Status 404 returned error can't find the container with id e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25
Oct 30 19:50:27.755455 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:27.755421    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7" event={"ID":"d3043364-7216-4a0d-83d9-ad302371c599","Type":"ContainerStarted","Data":"1c5a15f8fa368195b8e3e03d317cc5b6309772b1c97b9918e12977edbfea7eca"}
Oct 30 19:50:27.755846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:27.755472    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7" event={"ID":"d3043364-7216-4a0d-83d9-ad302371c599","Type":"ContainerStarted","Data":"e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25"}
Oct 30 19:50:27.795532 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:27.795484    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7" podStartSLOduration=1.79547199 podStartE2EDuration="1.79547199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:27.79213059 +0000 UTC m=+2775.827643054" watchObservedRunningTime="2026-10-30 19:50:27.79547199 +0000 UTC m=+2775.830984454"
Oct 30 19:50:28.645017 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:50:28.644978    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:50:28.766873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:28.766833    4320 generic.go:356] "Generic (PLEG): container finished" podID="d3043364-7216-4a0d-83d9-ad302371c599" containerID="1c5a15f8fa368195b8e3e03d317cc5b6309772b1c97b9918e12977edbfea7eca" exitCode=0
Oct 30 19:50:28.767290 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:28.766906    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7" event={"ID":"d3043364-7216-4a0d-83d9-ad302371c599","Type":"ContainerDied","Data":"1c5a15f8fa368195b8e3e03d317cc5b6309772b1c97b9918e12977edbfea7eca"}
Oct 30 19:50:29.937067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:29.937042    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:30.027362 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.027290    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\") pod \"d3043364-7216-4a0d-83d9-ad302371c599\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") "
Oct 30 19:50:30.027493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.027427    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\") pod \"d3043364-7216-4a0d-83d9-ad302371c599\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") "
Oct 30 19:50:30.027538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.027495    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\") pod \"d3043364-7216-4a0d-83d9-ad302371c599\" (UID: \"d3043364-7216-4a0d-83d9-ad302371c599\") "
Oct 30 19:50:30.027745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.027714    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca" pod "d3043364-7216-4a0d-83d9-ad302371c599" (UID: "d3043364-7216-4a0d-83d9-ad302371c599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:30.028151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.028125    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:30.029685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.029572    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca" pod "d3043364-7216-4a0d-83d9-ad302371c599" (UID: "d3043364-7216-4a0d-83d9-ad302371c599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:30.029685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.029632    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx" pod "d3043364-7216-4a0d-83d9-ad302371c599" (UID: "d3043364-7216-4a0d-83d9-ad302371c599"). InnerVolumeSpecName "kube-api-access-xrlbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:30.129234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.129207    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlbx\" (UniqueName: \"kubernetes.io/projected/d3043364-7216-4a0d-83d9-ad302371c599-kube-api-access-xrlbx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:30.129234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.129231    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3043364-7216-4a0d-83d9-ad302371c599-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:30.784974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.784930    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7" event={"ID":"d3043364-7216-4a0d-83d9-ad302371c599","Type":"ContainerDied","Data":"e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25"}
Oct 30 19:50:30.784974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.784971    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"
Oct 30 19:50:30.785190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:30.784997    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f91f12c349109bc14ce20ec94579bb8753f1e839b14f6a87cc6dc69037cc25"
Oct 30 19:50:42.480629 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.480581    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:50:42.481285 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.481268    4320 cpu_manager.go:414] "containerMap: removing container" podUID="d3043364-7216-4a0d-83d9-ad302371c599" containerName="revoke-keys"
Oct 30 19:50:42.481331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.481286    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3043364-7216-4a0d-83d9-ad302371c599" containerName="revoke-keys"
Oct 30 19:50:42.481404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.481394    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3043364-7216-4a0d-83d9-ad302371c599" containerName="revoke-keys"
Oct 30 19:50:42.485016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.484997    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.513380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.513332    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-852709-serving-cert\""
Oct 30 19:50:42.536622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.529297    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:50:42.612410 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.612370    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.612617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.612431    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.612714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.612692    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.612802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.612785    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjlp\" (UniqueName: \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.714375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.714332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.715269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.715243    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.716119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.716095    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.717319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.717285    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjlp\" (UniqueName: \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.717984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.717061    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.718692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.718671    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.725971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.725947    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.736528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.736465    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjlp\" (UniqueName: \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\") pod \"maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:42.816997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:42.816973    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:43.125571 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.125541    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:50:43.126528 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:50:43.126503    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf21005_97c1_49c6_a430_e737ba3eec10.slice/crio-20a2f811b11fa969755e1555420c480f8f2202169989a58247e66d8083c1bf6e WatchSource:0}: Error finding container 20a2f811b11fa969755e1555420c480f8f2202169989a58247e66d8083c1bf6e: Status 404 returned error can't find the container with id 20a2f811b11fa969755e1555420c480f8f2202169989a58247e66d8083c1bf6e
Oct 30 19:50:43.384746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.383276    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:50:43.384746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.383526    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" containerID="cri-o://c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8" gracePeriod=30
Oct 30 19:50:43.899073 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.899039    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerStarted","Data":"7d42436247554eb3c5f07d42eea057001e2ac490d11d791a9b9a72b76aaac6c9"}
Oct 30 19:50:43.899456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.899122    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:43.899456 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.899141    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerStarted","Data":"20a2f811b11fa969755e1555420c480f8f2202169989a58247e66d8083c1bf6e"}
Oct 30 19:50:43.942716 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:43.942667    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" podStartSLOduration=1.559407115 podStartE2EDuration="1.942651201s" totalImagesPullingTime="383.244086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.938781243 +0000 UTC m=+2791.974293706" watchObservedRunningTime="2026-10-30 19:50:43.942651201 +0000 UTC m=+2791.978163664"
Oct 30 19:50:48.109962 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.109923    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:50:48.110549 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.110022    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:50:48.723005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.722788    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:50:48.863066 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863028    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863080    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863166    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863214    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863242    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863287    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863402    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863437    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863571    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:48.863748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863580    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\") pod \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\" (UID: \"73b17461-7a83-4c4b-b51b-cc07a82c9aa5\") "
Oct 30 19:50:48.863748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863720    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:48.863932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.863902    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:48.864239 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864211    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:48.864636 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864613    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.864734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864641    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.864734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864655    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.864734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864668    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.864925 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.864781    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:48.867480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.867447    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:48.867935 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.867909    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:48.868042 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.868024    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:50:48.869036 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.869008    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb" pod "73b17461-7a83-4c4b-b51b-cc07a82c9aa5" (UID: "73b17461-7a83-4c4b-b51b-cc07a82c9aa5"). InnerVolumeSpecName "kube-api-access-dcrcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:48.966373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.966298    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.966373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.966325    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.966373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.966334    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrcb\" (UniqueName: \"kubernetes.io/projected/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-kube-api-access-dcrcb\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.966373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.966345    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:48.966373 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:48.966353    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b17461-7a83-4c4b-b51b-cc07a82c9aa5-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:50:49.031283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.031253    4320 generic.go:356] "Generic (PLEG): container finished" podID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerID="c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8" exitCode=0
Oct 30 19:50:49.031417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.031299    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"
Oct 30 19:50:49.031417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.031333    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" event={"ID":"73b17461-7a83-4c4b-b51b-cc07a82c9aa5","Type":"ContainerDied","Data":"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"}
Oct 30 19:50:49.031417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.031390    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9" event={"ID":"73b17461-7a83-4c4b-b51b-cc07a82c9aa5","Type":"ContainerDied","Data":"d4d2f5a65f805aceb46dfa1bfa05f26e17b0f643ca2718ee4dbd5f2279b1db6c"}
Oct 30 19:50:49.031417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.031414    4320 scope.go:122] "RemoveContainer" containerID="c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"
Oct 30 19:50:49.043831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.043810    4320 scope.go:122] "RemoveContainer" containerID="c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"
Oct 30 19:50:49.044098 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:50:49.044079    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8\": container with ID starting with c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8 not found: ID does not exist" containerID="c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"
Oct 30 19:50:49.044163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.044110    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8"} err="failed to get container status \"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8\": rpc error: code = NotFound desc = could not find container \"c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8\": container with ID starting with c23353a95a1b5a2a0555a5a4f274bb4e0d5edba7b07a6fb6e738bc1fd503b5d8 not found: ID does not exist"
Oct 30 19:50:49.087120 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.087093    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:50:49.090908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.090888    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-h8pr9"]
Oct 30 19:50:49.909711 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:49.909672    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.188:8443/health\": dial tcp 10.133.0.188:8443: connect: connection refused"
Oct 30 19:50:50.097997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:50.097955    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" path="/var/lib/kubelet/pods/73b17461-7a83-4c4b-b51b-cc07a82c9aa5/volumes"
Oct 30 19:50:54.215351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:54.213068    4320 generic.go:356] "Generic (PLEG): container finished" podID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerID="7d42436247554eb3c5f07d42eea057001e2ac490d11d791a9b9a72b76aaac6c9" exitCode=1
Oct 30 19:50:54.215351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:54.213193    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerDied","Data":"7d42436247554eb3c5f07d42eea057001e2ac490d11d791a9b9a72b76aaac6c9"}
Oct 30 19:50:54.215351 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:54.213791    4320 scope.go:122] "RemoveContainer" containerID="7d42436247554eb3c5f07d42eea057001e2ac490d11d791a9b9a72b76aaac6c9"
Oct 30 19:50:55.224001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:55.223965    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerStarted","Data":"1a47c61cf7c18ef8c9dc953836caed0497594d3b2958c3da454408b42279316e"}
Oct 30 19:50:55.224433 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:50:55.224185    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:50:58.645874 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:50:58.645835    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:51:00.029829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:00.029792    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"]
Oct 30 19:51:00.041227 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:00.041200    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63ba737-7qzjc"]
Oct 30 19:51:00.098234 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:00.098202    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa" path="/var/lib/kubelet/pods/fdfa3f0e-74e5-445a-9a4e-d9eed949a5aa/volumes"
Oct 30 19:51:01.236385 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:01.236358    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:51:11.033432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:11.033392    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"]
Oct 30 19:51:11.044829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:11.044799    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677830fa-67trg"]
Oct 30 19:51:12.098028 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:12.097994    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c36c531-792b-4aa3-b8a2-e08ba5d67a6d" path="/var/lib/kubelet/pods/2c36c531-792b-4aa3-b8a2-e08ba5d67a6d/volumes"
Oct 30 19:51:13.861157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:13.861124    4320 scope.go:122] "RemoveContainer" containerID="caa28354eedcdb13df9701de66fc505c65bc48b1032fe1912879e51f97fb3dd8"
Oct 30 19:51:13.871746 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:13.871716    4320 scope.go:122] "RemoveContainer" containerID="3df4dc6d2347a1f08106aea298a48514012fa4f956cc05caf910a3277793edaa"
Oct 30 19:51:13.881822 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:13.881803    4320 scope.go:122] "RemoveContainer" containerID="78dd6071974d1c812550bfaa685265f30740997632c2b3d79a200a8c3909ef68"
Oct 30 19:51:13.891458 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:13.891432    4320 scope.go:122] "RemoveContainer" containerID="51dd5949428dfac8ad75db532c72824d61bec85538e2dfbb907f2cc28de454e4"
Oct 30 19:51:15.027838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:15.027803    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"]
Oct 30 19:51:15.040233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:15.040203    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa8a8b-h7xv8"]
Oct 30 19:51:16.099712 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:16.099678    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492bf6de-78cb-4039-b875-a8b337cb255f" path="/var/lib/kubelet/pods/492bf6de-78cb-4039-b875-a8b337cb255f/volumes"
Oct 30 19:51:20.515053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:20.515021    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:51:24.036452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:24.036417    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"]
Oct 30 19:51:24.047714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:24.047681    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92bfd1f1-cx7g6"]
Oct 30 19:51:24.101306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:24.101273    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037b5c20-c7fb-4e9d-b75a-753a404b6cee" path="/var/lib/kubelet/pods/037b5c20-c7fb-4e9d-b75a-753a404b6cee/volumes"
Oct 30 19:51:28.644734 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:51:28.644698    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:51:35.537685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.537646    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerName="main" containerID="cri-o://de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b" gracePeriod=45
Oct 30 19:51:35.931971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.931942    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:51:35.995382 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995345    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.995650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995628    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.995767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995662    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:35.995767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995671    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\" (UniqueName: \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.995767 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995704    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.995958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995775    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.995958 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.995895    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\" (UniqueName: \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\") pod \"204d1aac-7a6c-4950-9632-86ea45fb8f48\" (UID: \"204d1aac-7a6c-4950-9632-86ea45fb8f48\") "
Oct 30 19:51:35.996690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.996299    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:35.996690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.996337    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:35.996690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.996580    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:35.996690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.996641    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:35.996690 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.996660    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:35.998702 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.998682    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:35.999134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.999108    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "kube-api-access-j7fxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:35.999380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:35.999347    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm" pod "204d1aac-7a6c-4950-9632-86ea45fb8f48" (UID: "204d1aac-7a6c-4950-9632-86ea45fb8f48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:36.033719 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.033688    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"]
Oct 30 19:51:36.036858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.036836    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e402df-6s5fn"]
Oct 30 19:51:36.097661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.097636    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6d52d7-698b-48ae-89e8-53ab9c061af9" path="/var/lib/kubelet/pods/3e6d52d7-698b-48ae-89e8-53ab9c061af9/volumes"
Oct 30 19:51:36.097774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.097719    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204d1aac-7a6c-4950-9632-86ea45fb8f48-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:36.097774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.097740    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fxg\" (UniqueName: \"kubernetes.io/projected/204d1aac-7a6c-4950-9632-86ea45fb8f48-kube-api-access-j7fxg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:36.097774 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.097750    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204d1aac-7a6c-4950-9632-86ea45fb8f48-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:51:36.576472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.576431    4320 generic.go:356] "Generic (PLEG): container finished" podID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerID="de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b" exitCode=0
Oct 30 19:51:36.576988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.576473    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"
Oct 30 19:51:36.576988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.576557    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" event={"ID":"204d1aac-7a6c-4950-9632-86ea45fb8f48","Type":"ContainerDied","Data":"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"}
Oct 30 19:51:36.576988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.576652    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" event={"ID":"204d1aac-7a6c-4950-9632-86ea45fb8f48","Type":"ContainerDied","Data":"5d7f72e574fda6c96deb4fcc45dc2cd4750b937589be053a12736f2a87249dda"}
Oct 30 19:51:36.576988 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.576682    4320 scope.go:122] "RemoveContainer" containerID="de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"
Oct 30 19:51:36.588007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.587991    4320 scope.go:122] "RemoveContainer" containerID="de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"
Oct 30 19:51:36.588309 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:51:36.588290    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b\": container with ID starting with de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b not found: ID does not exist" containerID="de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"
Oct 30 19:51:36.588349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.588317    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b"} err="failed to get container status \"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b\": rpc error: code = NotFound desc = could not find container \"de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b\": container with ID starting with de6af313e9491befc68c17c43588f56315d07adf5b6c6dc978e8167ccb00ab2b not found: ID does not exist"
Oct 30 19:51:36.630339 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.630311    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:51:36.641434 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.641407    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc"]
Oct 30 19:51:36.709947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:36.709920    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-6b7ccc/sub-test-model-6b7ccc-kserve-8656454977-w9vdc" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.175:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:38.104699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:38.104662    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" path="/var/lib/kubelet/pods/204d1aac-7a6c-4950-9632-86ea45fb8f48/volumes"
Oct 30 19:51:38.357674 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:38.348431    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:51:40.035865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:40.035816    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"]
Oct 30 19:51:40.049250 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:40.049202    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-241aecd4-56pqn"]
Oct 30 19:51:40.173622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:40.167569    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e" path="/var/lib/kubelet/pods/b0e2b82f-8f2d-427b-b27d-81cba5a6cc4e/volumes"
Oct 30 19:51:47.036487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:47.036452    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"]
Oct 30 19:51:47.039743 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:47.039721    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9cbea9-7wqbp"]
Oct 30 19:51:48.100523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:48.100486    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32663bd8-b235-4b24-880b-8ce0081fb228" path="/var/lib/kubelet/pods/32663bd8-b235-4b24-880b-8ce0081fb228/volumes"
Oct 30 19:51:57.948865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.948831    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:51:57.949734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949514    4320 cpu_manager.go:414] "containerMap: removing container" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy"
Oct 30 19:51:57.949734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949530    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy"
Oct 30 19:51:57.949734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949563    4320 cpu_manager.go:414] "containerMap: removing container" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerName="main"
Oct 30 19:51:57.949734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949568    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerName="main"
Oct 30 19:51:57.949734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949733    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b17461-7a83-4c4b-b51b-cc07a82c9aa5" containerName="istio-proxy"
Oct 30 19:51:57.949974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.949751    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="204d1aac-7a6c-4950-9632-86ea45fb8f48" containerName="main"
Oct 30 19:51:57.953825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.953805    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:57.974699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:57.974664    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:51:58.028440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.028414    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"]
Oct 30 19:51:58.031621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.031586    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3caa79e2-x6hfz"]
Oct 30 19:51:58.076511 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076480    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076521    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076542    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076662 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076643    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076705    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn98\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076916 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076885    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.076977 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.076946    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.077037 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.077016    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.077097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.077074    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.099076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.099044    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103f160e-141a-4d5e-8f7e-2a005e5343e0" path="/var/lib/kubelet/pods/103f160e-141a-4d5e-8f7e-2a005e5343e0/volumes"
Oct 30 19:51:58.178235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178206    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178255    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178300    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178332    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178385    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178409    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178429    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178465    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178614    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178692    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn98\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178706    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.178983 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.178924    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.179103 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.179042    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.179222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.179084    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.180875 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.180854    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.181127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.181104    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.197483 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.197457    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.197605 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.197551    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn98\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.267641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.267568    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:58.604342 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.604318    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:51:58.605489 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:51:58.605465    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5715dc59_ae45_49d0_a249_de008c9765b0.slice/crio-d8f3dfafdfdc1dc3c6607e9601cbf748bb4785c169136f2bcb17daff36ff9cd3 WatchSource:0}: Error finding container d8f3dfafdfdc1dc3c6607e9601cbf748bb4785c169136f2bcb17daff36ff9cd3: Status 404 returned error can't find the container with id d8f3dfafdfdc1dc3c6607e9601cbf748bb4785c169136f2bcb17daff36ff9cd3
Oct 30 19:51:58.608308 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.608273    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:51:58.608402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.608341    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:51:58.608402 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.608370    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:51:58.646327 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:51:58.646300    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:51:58.833473 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.833434    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" event={"ID":"5715dc59-ae45-49d0-a249-de008c9765b0","Type":"ContainerStarted","Data":"3b5a6097c85ead176d2f93cab0cd9a9e91af2b9e4498d7211fc7a2def01a480e"}
Oct 30 19:51:58.833705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.833499    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" event={"ID":"5715dc59-ae45-49d0-a249-de008c9765b0","Type":"ContainerStarted","Data":"d8f3dfafdfdc1dc3c6607e9601cbf748bb4785c169136f2bcb17daff36ff9cd3"}
Oct 30 19:51:58.876723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:58.876629    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podStartSLOduration=1.876616142 podStartE2EDuration="1.876616142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:58.873459851 +0000 UTC m=+2866.908972317" watchObservedRunningTime="2026-10-30 19:51:58.876616142 +0000 UTC m=+2866.912128598"
Oct 30 19:51:59.268581 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:59.268550    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:51:59.291622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:59.284987    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:51:59.291622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:51:59.285050    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:00.268216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:00.268183    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:52:00.268387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:00.268231    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:01.268095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:01.268060    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:52:01.268472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:01.268118    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:02.268591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:02.268555    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:52:02.285023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:02.284644    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:03.268468 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:03.268436    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:52:03.268678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:03.268492    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:04.267814 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:04.267785    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:52:04.267992 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:04.267831    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 19:52:05.277059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:05.277025    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:52:05.278137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:05.278076    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:52:05.279164 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:05.279140    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:52:13.941311 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:13.941280    4320 scope.go:122] "RemoveContainer" containerID="17b6536431f59aa60460c1ff3c627d750050747a4c7f4a9682b762acd9480426"
Oct 30 19:52:13.951232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:13.951210    4320 scope.go:122] "RemoveContainer" containerID="02b67584ada7d08dee14ae3a0386a0f7323a432b8a16d0cf493287725bcc5a2b"
Oct 30 19:52:13.960409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:13.960389    4320 scope.go:122] "RemoveContainer" containerID="5565cfaabf3b6463552e80aab406ee4c013084cfab026f95c06f2aa5b14eea15"
Oct 30 19:52:28.646460 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:52:28.646418    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:52:46.178989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.178957    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 19:52:46.183659 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.183637    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.205019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.204974    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ed2ede-serving-cert\""
Oct 30 19:52:46.223307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.223282    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 19:52:46.336560 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.336526    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.336740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.336688    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cn7\" (UniqueName: \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.336865 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.336832    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.336908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.336899    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.437757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.437681    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.437757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.437730    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.437959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.437804    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.437959 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.437874    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cn7\" (UniqueName: \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.438460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.438429    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.440349 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.440327    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.440464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.440367    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.457429 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.457401    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cn7\" (UniqueName: \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\") pod \"maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.503963 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.503676    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:46.834019 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:46.833985    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 19:52:46.835236 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:52:46.835209    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198b2307_c817_4ed3_b88e_7d07b3383357.slice/crio-a030ec5caf97702757969684770524693d7bdaeec738b445a827f21310469167 WatchSource:0}: Error finding container a030ec5caf97702757969684770524693d7bdaeec738b445a827f21310469167: Status 404 returned error can't find the container with id a030ec5caf97702757969684770524693d7bdaeec738b445a827f21310469167
Oct 30 19:52:47.386337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:47.386299    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" event={"ID":"198b2307-c817-4ed3-b88e-7d07b3383357","Type":"ContainerStarted","Data":"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"}
Oct 30 19:52:47.386802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:47.386358    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" event={"ID":"198b2307-c817-4ed3-b88e-7d07b3383357","Type":"ContainerStarted","Data":"a030ec5caf97702757969684770524693d7bdaeec738b445a827f21310469167"}
Oct 30 19:52:47.386802 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:47.386386    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:47.450671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:47.443451    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" podStartSLOduration=1.086225098 podStartE2EDuration="1.443428163s" totalImagesPullingTime="357.203065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:47.429583794 +0000 UTC m=+2915.465096262" watchObservedRunningTime="2026-10-30 19:52:47.443428163 +0000 UTC m=+2915.478940628"
Oct 30 19:52:53.400054 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:52:53.400026    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 19:52:58.645867 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:52:58.645774    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:53:10.742255 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.742216    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:53:10.748148 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.748112    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.765309 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.765282    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:53:10.772463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.772436    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b36ad\""
Oct 30 19:53:10.772638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.772579    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b36ad-dockercfg-2fsms\""
Oct 30 19:53:10.773097 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.773075    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.773193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.773140    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8hf\" (UniqueName: \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.875995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.874704    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.875995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.874779    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8hf\" (UniqueName: \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.875995 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.875948    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.897739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.897391    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8hf\" (UniqueName: \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\") pod \"payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:10.978024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:10.977988    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:53:11.066176 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.066087    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:11.449231 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.449195    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:53:11.453787 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:53:11.453760    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f7fabd_5a75_4620_bbac_1486fcd15d6a.slice/crio-b267c90bdebe95c808ad06cca004c95954876564cc5300cc8426564a831eb0c0 WatchSource:0}: Error finding container b267c90bdebe95c808ad06cca004c95954876564cc5300cc8426564a831eb0c0: Status 404 returned error can't find the container with id b267c90bdebe95c808ad06cca004c95954876564cc5300cc8426564a831eb0c0
Oct 30 19:53:11.598667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.598581    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" event={"ID":"d6f7fabd-5a75-4620-bbac-1486fcd15d6a","Type":"ContainerStarted","Data":"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"}
Oct 30 19:53:11.598667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.598667    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:11.598869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.598687    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" event={"ID":"d6f7fabd-5a75-4620-bbac-1486fcd15d6a","Type":"ContainerStarted","Data":"b267c90bdebe95c808ad06cca004c95954876564cc5300cc8426564a831eb0c0"}
Oct 30 19:53:11.651226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:11.651160    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" podStartSLOduration=1.651137876 podStartE2EDuration="1.651137876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.646802698 +0000 UTC m=+2939.682315185" watchObservedRunningTime="2026-10-30 19:53:11.651137876 +0000 UTC m=+2939.686650341"
Oct 30 19:53:14.012587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.012550    4320 scope.go:122] "RemoveContainer" containerID="d314c6888cb06c8b2f05dfe81130488120b3dbe20e98cf319a7851fb4f54c422"
Oct 30 19:53:14.024440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.024415    4320 scope.go:122] "RemoveContainer" containerID="8cb088c85834326e3d5e460fb47ddfea31f6b218f98e9f8ea8a749f1ded23db4"
Oct 30 19:53:14.036380 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.036354    4320 scope.go:122] "RemoveContainer" containerID="376af8768db6c9e7a76c09e7ad5c13b405b61cc97b4a741fd1b801fde537af81"
Oct 30 19:53:14.045440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.045419    4320 scope.go:122] "RemoveContainer" containerID="a2ec8de629585ec08f54e02d59475975d875626f46d6bc9b3069042dcf6993a5"
Oct 30 19:53:14.054035 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.054015    4320 scope.go:122] "RemoveContainer" containerID="b0543e4c734b85ee0e4c3061cbe5b2bb2614f2d8b6e3d599f07f935c363c78ac"
Oct 30 19:53:14.062244 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:14.062225    4320 scope.go:122] "RemoveContainer" containerID="dde9bb00783832b348ac3f61545397a5c154bbc38cddd7ea6fed64c709127271"
Oct 30 19:53:22.637794 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:22.637766    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:53:28.645719 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:53:28.645679    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:53:29.036049 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:29.035971    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"]
Oct 30 19:53:29.047185 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:29.047161    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6aad54-xsb2z"]
Oct 30 19:53:30.099724 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:53:30.099690    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7381152e-2c68-41ef-9cbb-5fccf02aa6f1" path="/var/lib/kubelet/pods/7381152e-2c68-41ef-9cbb-5fccf02aa6f1/volumes"
Oct 30 19:53:58.644318 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:53:58.644282    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:54:12.522619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:12.521834    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:54:14.119192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:14.119157    4320 scope.go:122] "RemoveContainer" containerID="12cf8212fe679b006cb998b69582bc7f4d92f9984fa7200a3bc3bd4b5e66c83b"
Oct 30 19:54:14.129459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:14.129438    4320 scope.go:122] "RemoveContainer" containerID="6e4470c40b62ab0163d33131c01b4f0516be2eaae61f7f3a0071e4279d6c979b"
Oct 30 19:54:14.138839 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:14.138819    4320 scope.go:122] "RemoveContainer" containerID="eb6063d12547ea9794fb93f39c8c295e34f17d394522422b1673a323fc203620"
Oct 30 19:54:22.150965 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.150934    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:54:22.151768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.151712    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api" containerID="cri-o://1a47c61cf7c18ef8c9dc953836caed0497594d3b2958c3da454408b42279316e" gracePeriod=30
Oct 30 19:54:22.407563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.407471    4320 generic.go:356] "Generic (PLEG): container finished" podID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerID="1a47c61cf7c18ef8c9dc953836caed0497594d3b2958c3da454408b42279316e" exitCode=0
Oct 30 19:54:22.408706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.408673    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerDied","Data":"1a47c61cf7c18ef8c9dc953836caed0497594d3b2958c3da454408b42279316e"}
Oct 30 19:54:22.408859 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.408753    4320 scope.go:122] "RemoveContainer" containerID="7d42436247554eb3c5f07d42eea057001e2ac490d11d791a9b9a72b76aaac6c9"
Oct 30 19:54:22.480216 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.480189    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:54:22.530127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.530101    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\") pod \"4bf21005-97c1-49c6-a430-e737ba3eec10\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") "
Oct 30 19:54:22.530288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.530143    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\" (UniqueName: \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\") pod \"4bf21005-97c1-49c6-a430-e737ba3eec10\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") "
Oct 30 19:54:22.530288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.530196    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\") pod \"4bf21005-97c1-49c6-a430-e737ba3eec10\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") "
Oct 30 19:54:22.530428 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.530332    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\") pod \"4bf21005-97c1-49c6-a430-e737ba3eec10\" (UID: \"4bf21005-97c1-49c6-a430-e737ba3eec10\") "
Oct 30 19:54:22.530657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.530626    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca" pod "4bf21005-97c1-49c6-a430-e737ba3eec10" (UID: "4bf21005-97c1-49c6-a430-e737ba3eec10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:22.531069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.531042    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf21005-97c1-49c6-a430-e737ba3eec10-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:22.532951 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.532918    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs" pod "4bf21005-97c1-49c6-a430-e737ba3eec10" (UID: "4bf21005-97c1-49c6-a430-e737ba3eec10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:22.533044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.532924    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp" pod "4bf21005-97c1-49c6-a430-e737ba3eec10" (UID: "4bf21005-97c1-49c6-a430-e737ba3eec10"). InnerVolumeSpecName "kube-api-access-vtjlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:22.534238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.533769    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls" pod "4bf21005-97c1-49c6-a430-e737ba3eec10" (UID: "4bf21005-97c1-49c6-a430-e737ba3eec10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:22.666296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.657291    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:22.666296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.657329    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjlp\" (UniqueName: \"kubernetes.io/projected/4bf21005-97c1-49c6-a430-e737ba3eec10-kube-api-access-vtjlp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:22.666296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:22.657344    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf21005-97c1-49c6-a430-e737ba3eec10-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:23.417798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:23.417771    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"
Oct 30 19:54:23.418249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:23.417820    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x" event={"ID":"4bf21005-97c1-49c6-a430-e737ba3eec10","Type":"ContainerDied","Data":"20a2f811b11fa969755e1555420c480f8f2202169989a58247e66d8083c1bf6e"}
Oct 30 19:54:23.418249 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:23.417887    4320 scope.go:122] "RemoveContainer" containerID="1a47c61cf7c18ef8c9dc953836caed0497594d3b2958c3da454408b42279316e"
Oct 30 19:54:23.469680 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:23.469659    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:54:23.481521 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:23.481494    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-852709-6777bf8bfc-l9g2x"]
Oct 30 19:54:24.098334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:24.098301    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" path="/var/lib/kubelet/pods/4bf21005-97c1-49c6-a430-e737ba3eec10/volumes"
Oct 30 19:54:28.645707 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:54:28.645669    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:54:33.420621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.419734    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 19:54:33.441620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.431080    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:33.441620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.431105    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:33.441620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.431337    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:33.441620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.431358    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:33.495954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.495921    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.544714 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.539879    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2afc11-openshift-default-dockercfg-6ggw5\""
Oct 30 19:54:33.628696 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.628654    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.650236    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.650330    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.650694    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzfr\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.650811    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.651073    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.651140    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.651399    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.651431    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.652723 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.651656    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.755683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755541    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.755868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755748    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.755868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755807    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.755868 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755848    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzfr\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755944    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.755991    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756044    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756135    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756164    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756178    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756479    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.756634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.756615    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.757031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.757011    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.757118 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.757104    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.758492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.758451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.758788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.758765    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.775021 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.774999    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.775805 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.775781    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzfr\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\") pod \"e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") " pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:33.929587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:33.929554    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:34.029735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.029652    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"]
Oct 30 19:54:34.033192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.033157    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8005804d-mrfgp"]
Oct 30 19:54:34.097900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.097868    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0806de01-f769-4cfc-8f34-2892a9c7219a" path="/var/lib/kubelet/pods/0806de01-f769-4cfc-8f34-2892a9c7219a/volumes"
Oct 30 19:54:34.258775 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.258743    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 19:54:34.259852 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:54:34.259821    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff18f27f_6755_4e38_b9cc_2879a4d17529.slice/crio-bc24c66d4fba1870594cdcf1eee88c9eb2ce9948e565c5268cd64bcf77c3d6ce WatchSource:0}: Error finding container bc24c66d4fba1870594cdcf1eee88c9eb2ce9948e565c5268cd64bcf77c3d6ce: Status 404 returned error can't find the container with id bc24c66d4fba1870594cdcf1eee88c9eb2ce9948e565c5268cd64bcf77c3d6ce
Oct 30 19:54:34.263566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.263520    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:34.263669 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.263633    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:34.263744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.263677    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:34.508463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.508424    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerStarted","Data":"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"}
Oct 30 19:54:34.508876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.508476    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerStarted","Data":"bc24c66d4fba1870594cdcf1eee88c9eb2ce9948e565c5268cd64bcf77c3d6ce"}
Oct 30 19:54:34.554726 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.554635    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podStartSLOduration=1.5546185879999999 podStartE2EDuration="1.554618588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:34.55082718 +0000 UTC m=+3022.586339666" watchObservedRunningTime="2026-10-30 19:54:34.554618588 +0000 UTC m=+3022.590131089"
Oct 30 19:54:34.929860 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:34.929830    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:35.936472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:35.936444    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:36.069431 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.069374    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:54:36.070076 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.070062    4320 cpu_manager.go:414] "containerMap: removing container" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:36.070154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.070078    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf21005-97c1-49c6-a430-e737ba3eec10" containerName="maas-api"
Oct 30 19:54:36.074116 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.074097    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.090496 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.090371    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5250ee-openshift-default-dockercfg-bzstj\""
Oct 30 19:54:36.101913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.101887    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:54:36.199279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199210    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199408 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199338    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199408 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199377    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199509    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wstf\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199548    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199618 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199570    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199747    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199838 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199802    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.199923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.199906    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301230    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301293    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301478 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301398    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301536 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301509    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301617 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301573    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301621    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301668    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wstf\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301709    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301795 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301742    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.301904 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.301865    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.302063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.302038    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.302140 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.302124    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.302319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.302298    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.302397 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.302375    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.303706 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.303683    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.304566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.304533    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.320119 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.320098    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.320384 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.320353    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wstf\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.389733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.389707    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:36.522561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.522533    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:36.523621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.523578    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:54:36.710635 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.710610    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:54:36.711902 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:54:36.711873    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc5bed6_279f_4180_b9de_9403b9b5f1b9.slice/crio-da179d3f5a74cb2662e90b5f79880d215e34add69ec8185c8f6b0b9ddfa89877 WatchSource:0}: Error finding container da179d3f5a74cb2662e90b5f79880d215e34add69ec8185c8f6b0b9ddfa89877: Status 404 returned error can't find the container with id da179d3f5a74cb2662e90b5f79880d215e34add69ec8185c8f6b0b9ddfa89877
Oct 30 19:54:36.714961 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.714924    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:36.715067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.714997    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:36.715067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:36.715026    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:37.532354 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:37.532310    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" event={"ID":"0cc5bed6-279f-4180-b9de-9403b9b5f1b9","Type":"ContainerStarted","Data":"d3af5d873325b94346ba29a7ae049832b31f33ceb15016f4453b495c04e0a848"}
Oct 30 19:54:37.532810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:37.532423    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" event={"ID":"0cc5bed6-279f-4180-b9de-9403b9b5f1b9","Type":"ContainerStarted","Data":"da179d3f5a74cb2662e90b5f79880d215e34add69ec8185c8f6b0b9ddfa89877"}
Oct 30 19:54:37.613620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:37.604326    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podStartSLOduration=1.604302203 podStartE2EDuration="1.604302203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:37.574914065 +0000 UTC m=+3025.610426530" watchObservedRunningTime="2026-10-30 19:54:37.604302203 +0000 UTC m=+3025.639814667"
Oct 30 19:54:38.393847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:38.393810    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:38.395307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:38.395279    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:38.395460 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:38.395333    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:39.392621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:39.390974    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:39.392621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:39.391041    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:40.406650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:40.394427    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:40.406650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:40.394519    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:41.394768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:41.394730    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:41.395085 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:41.395046    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:41.398518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:41.398494    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:54:42.330756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.330721    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:54:42.348488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.348382    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:54:42.348666 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.348498    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.364071 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.363960    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-59f9ed-openshift-default-dockercfg-2ckjb\""
Oct 30 19:54:42.406771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.406735    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:42.407027 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.406998    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.523916    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524015    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524042    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524083    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524117    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524188    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524210    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524242    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.538650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.524312    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcnk\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.666672 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.630021    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.685275 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.685224    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcnk\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.685755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.685726    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686024 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686003    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686146    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686278    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686417 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686400    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686610 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686579    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.686733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.686716    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.698479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.698401    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.699048 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.699029    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.699557 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.699536    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.702740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.702718    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.703151 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.703114    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.710665 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.707648    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.710810 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.707879    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.744550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.744515    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.758345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.758301    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcnk\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\") pod \"e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") " pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:42.992887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:42.992801    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:43.340641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.340606    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:54:43.341088 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:54:43.340690    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f7ad27_04e8_432d_ac33_c80fb8309095.slice/crio-984942d80187ca877943b8c000204afd6e67e272dedfc786c54efb79651364ed WatchSource:0}: Error finding container 984942d80187ca877943b8c000204afd6e67e272dedfc786c54efb79651364ed: Status 404 returned error can't find the container with id 984942d80187ca877943b8c000204afd6e67e272dedfc786c54efb79651364ed
Oct 30 19:54:43.344741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.344702    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:43.344854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.344798    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:43.344854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.344844    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:54:43.390776 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.390750    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:54:43.390882 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.390808    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:54:43.719281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.719232    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerStarted","Data":"bfb769ea5d07bf565958d71d249c4417ad2b6303290646d371595dc6a0d7c537"}
Oct 30 19:54:43.719477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.719302    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerStarted","Data":"984942d80187ca877943b8c000204afd6e67e272dedfc786c54efb79651364ed"}
Oct 30 19:54:43.765733 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.765502    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podStartSLOduration=1.765484486 podStartE2EDuration="1.765484486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:43.762705542 +0000 UTC m=+3031.798218006" watchObservedRunningTime="2026-10-30 19:54:43.765484486 +0000 UTC m=+3031.800996952"
Oct 30 19:54:43.993649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:43.993547    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:44.032241 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.032205    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"]
Oct 30 19:54:44.035501 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.035478    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fada187-4p68z"]
Oct 30 19:54:44.101273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.101233    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468bab8e-aad7-4227-b693-9952d9166a50" path="/var/lib/kubelet/pods/468bab8e-aad7-4227-b693-9952d9166a50/volumes"
Oct 30 19:54:44.395591 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.395557    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:44.396080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.396002    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:44.396764 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:44.396735    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:54:45.002160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:45.002128    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:45.735094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:45.735057    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:45.736075 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:45.736056    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:54:46.399328 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.399269    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" podUID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" containerName="payload-pre-processing" containerID="cri-o://bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a" gracePeriod=25
Oct 30 19:54:46.677337 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.677312    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:54:46.748298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.748259    4320 generic.go:356] "Generic (PLEG): container finished" podID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" containerID="bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a" exitCode=0
Oct 30 19:54:46.748298 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.748302    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"
Oct 30 19:54:46.748957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.748342    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" event={"ID":"d6f7fabd-5a75-4620-bbac-1486fcd15d6a","Type":"ContainerDied","Data":"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"}
Oct 30 19:54:46.748957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.748456    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx" event={"ID":"d6f7fabd-5a75-4620-bbac-1486fcd15d6a","Type":"ContainerDied","Data":"b267c90bdebe95c808ad06cca004c95954876564cc5300cc8426564a831eb0c0"}
Oct 30 19:54:46.748957 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.748498    4320 scope.go:122] "RemoveContainer" containerID="bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"
Oct 30 19:54:46.763924 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.763903    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\" (UniqueName: \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\") pod \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") "
Oct 30 19:54:46.764124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.764107    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\") pod \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\" (UID: \"d6f7fabd-5a75-4620-bbac-1486fcd15d6a\") "
Oct 30 19:54:46.764519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.764495    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume" pod "d6f7fabd-5a75-4620-bbac-1486fcd15d6a" (UID: "d6f7fabd-5a75-4620-bbac-1486fcd15d6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:46.764817 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.764798    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:46.766190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.766168    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf" pod "d6f7fabd-5a75-4620-bbac-1486fcd15d6a" (UID: "d6f7fabd-5a75-4620-bbac-1486fcd15d6a"). InnerVolumeSpecName "kube-api-access-xk8hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:46.773015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.769786    4320 scope.go:122] "RemoveContainer" containerID="bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"
Oct 30 19:54:46.773015 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:54:46.772847    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a\": container with ID starting with bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a not found: ID does not exist" containerID="bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"
Oct 30 19:54:46.773015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.772880    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a"} err="failed to get container status \"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a\": rpc error: code = NotFound desc = could not find container \"bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a\": container with ID starting with bce41459871298d28e6a59d8f3bdfe341509fceb60fc37f0295c5b7aa2d5439a not found: ID does not exist"
Oct 30 19:54:46.866791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:46.866762    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8hf\" (UniqueName: \"kubernetes.io/projected/d6f7fabd-5a75-4620-bbac-1486fcd15d6a-kube-api-access-xk8hf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:54:47.101893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:47.101860    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:54:47.111678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:47.111646    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b36ad-5744cb9895-z8jvx"]
Oct 30 19:54:48.291830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:54:48.287135    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" path="/var/lib/kubelet/pods/d6f7fabd-5a75-4620-bbac-1486fcd15d6a/volumes"
Oct 30 19:54:58.645283 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:54:58.645244    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:55:01.904095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:01.904061    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:55:13.215691 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:13.215654    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:55:13.216278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:13.215884    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" podUID="55f66ef3-600e-4a16-b61d-f255bb0b943c" containerName="maas-api" containerID="cri-o://14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402" gracePeriod=30
Oct 30 19:55:13.948623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:13.948350    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:55:14.047888 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.047835    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\") pod \"55f66ef3-600e-4a16-b61d-f255bb0b943c\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") "
Oct 30 19:55:14.048090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.047888    4320 generic.go:356] "Generic (PLEG): container finished" podID="55f66ef3-600e-4a16-b61d-f255bb0b943c" containerID="14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402" exitCode=0
Oct 30 19:55:14.048090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.047964    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" event={"ID":"55f66ef3-600e-4a16-b61d-f255bb0b943c","Type":"ContainerDied","Data":"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"}
Oct 30 19:55:14.048090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.047990    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") pod \"55f66ef3-600e-4a16-b61d-f255bb0b943c\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") "
Oct 30 19:55:14.048281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048125    4320 scope.go:122] "RemoveContainer" containerID="14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"
Oct 30 19:55:14.048281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048180    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\") pod \"55f66ef3-600e-4a16-b61d-f255bb0b943c\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") "
Oct 30 19:55:14.048281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048220    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\") pod \"55f66ef3-600e-4a16-b61d-f255bb0b943c\" (UID: \"55f66ef3-600e-4a16-b61d-f255bb0b943c\") "
Oct 30 19:55:14.048463 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048441    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"
Oct 30 19:55:14.048525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048506    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca" pod "55f66ef3-600e-4a16-b61d-f255bb0b943c" (UID: "55f66ef3-600e-4a16-b61d-f255bb0b943c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:14.048831 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048051    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f" event={"ID":"55f66ef3-600e-4a16-b61d-f255bb0b943c","Type":"ContainerDied","Data":"830f3217fccbaa614c0d4c45fe6125e1db4dbd2c4b6da28b180bab401d3522bf"}
Oct 30 19:55:14.048968 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.048891    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f66ef3-600e-4a16-b61d-f255bb0b943c-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:55:14.050870 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.050842    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls" pod "55f66ef3-600e-4a16-b61d-f255bb0b943c" (UID: "55f66ef3-600e-4a16-b61d-f255bb0b943c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:14.051004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.050978    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj" pod "55f66ef3-600e-4a16-b61d-f255bb0b943c" (UID: "55f66ef3-600e-4a16-b61d-f255bb0b943c"). InnerVolumeSpecName "kube-api-access-cbjhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:14.051113 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.051096    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs" pod "55f66ef3-600e-4a16-b61d-f255bb0b943c" (UID: "55f66ef3-600e-4a16-b61d-f255bb0b943c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:14.059086 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.059063    4320 scope.go:122] "RemoveContainer" containerID="14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"
Oct 30 19:55:14.059362 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:55:14.059342    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402\": container with ID starting with 14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402 not found: ID does not exist" containerID="14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"
Oct 30 19:55:14.059436 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.059373    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402"} err="failed to get container status \"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402\": rpc error: code = NotFound desc = could not find container \"14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402\": container with ID starting with 14017e730513a71d6832b3f369028dd1605186c40b7cef41cc438d411bd38402 not found: ID does not exist"
Oct 30 19:55:14.151283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.151253    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:55:14.151283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.151280    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjhj\" (UniqueName: \"kubernetes.io/projected/55f66ef3-600e-4a16-b61d-f255bb0b943c-kube-api-access-cbjhj\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:55:14.151283 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.151289    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f66ef3-600e-4a16-b61d-f255bb0b943c-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:55:14.179265 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.179240    4320 scope.go:122] "RemoveContainer" containerID="439a1dd00837c45e9aa981a57236e7c3e69f84f8e6175d275c8493624cd7f4b3"
Oct 30 19:55:14.188885 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.188865    4320 scope.go:122] "RemoveContainer" containerID="a401ae3b55bdf8a9d9c29caa30c6dd6070443e30f15ba19641fee8cebd9a07a2"
Oct 30 19:55:14.390045 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.390018    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:55:14.401553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:14.401483    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7b7bc-56b7fcc478-x6c5f"]
Oct 30 19:55:16.097090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:16.097054    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f66ef3-600e-4a16-b61d-f255bb0b943c" path="/var/lib/kubelet/pods/55f66ef3-600e-4a16-b61d-f255bb0b943c/volumes"
Oct 30 19:55:20.029252 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.029218    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"]
Oct 30 19:55:20.040004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.039975    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c2283c-ppjz2"]
Oct 30 19:55:20.098157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.098129    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffac3574-413f-48ec-8df9-7d6723a20d1c" path="/var/lib/kubelet/pods/ffac3574-413f-48ec-8df9-7d6723a20d1c/volumes"
Oct 30 19:55:20.940964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.940932    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 19:55:20.941961 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.941941    4320 cpu_manager.go:414] "containerMap: removing container" podUID="55f66ef3-600e-4a16-b61d-f255bb0b943c" containerName="maas-api"
Oct 30 19:55:20.942022 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.941965    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f66ef3-600e-4a16-b61d-f255bb0b943c" containerName="maas-api"
Oct 30 19:55:20.942060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.942021    4320 cpu_manager.go:414] "containerMap: removing container" podUID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" containerName="payload-pre-processing"
Oct 30 19:55:20.942060 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.942030    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" containerName="payload-pre-processing"
Oct 30 19:55:20.942218 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.942207    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f7fabd-5a75-4620-bbac-1486fcd15d6a" containerName="payload-pre-processing"
Oct 30 19:55:20.942254 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.942232    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f66ef3-600e-4a16-b61d-f255bb0b943c" containerName="maas-api"
Oct 30 19:55:20.949593 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.949569    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:20.970949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.970925    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2afc11-serving-cert\""
Oct 30 19:55:20.985294 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:20.985268    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 19:55:21.090179 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.090154    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.090543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.090258    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.090543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.090355    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4wh\" (UniqueName: \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.090543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.090509    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.195414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.195312    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.195972 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.195937    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4wh\" (UniqueName: \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.196277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.196261    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.196448 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.196422    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.205692 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.205644    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.206567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.206540    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.207409 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.207144    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.219717 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.219634    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4wh\" (UniqueName: \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\") pod \"maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.269558 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.269535    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:21.579053 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.579025    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 19:55:21.580052 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:55:21.580026    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14eb7a09_96f9_4a4e_8722_083fa758dd54.slice/crio-1c5ae85532458c934ea21df9cd298498705bc3f37af39e52c72b22d79ab55c5c WatchSource:0}: Error finding container 1c5ae85532458c934ea21df9cd298498705bc3f37af39e52c72b22d79ab55c5c: Status 404 returned error can't find the container with id 1c5ae85532458c934ea21df9cd298498705bc3f37af39e52c72b22d79ab55c5c
Oct 30 19:55:21.581528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:21.581507    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:55:22.250675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:22.249867    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" event={"ID":"14eb7a09-96f9-4a4e-8722-083fa758dd54","Type":"ContainerStarted","Data":"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"}
Oct 30 19:55:22.250675 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:22.249962    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" event={"ID":"14eb7a09-96f9-4a4e-8722-083fa758dd54","Type":"ContainerStarted","Data":"1c5ae85532458c934ea21df9cd298498705bc3f37af39e52c72b22d79ab55c5c"}
Oct 30 19:55:22.257620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:22.256426    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:28.647883 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:55:28.647825    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:55:29.268482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:29.268455    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 19:55:29.311094 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:29.311034    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" podStartSLOduration=8.931137454 podStartE2EDuration="9.311012174s" totalImagesPullingTime="379.87472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:22.314309815 +0000 UTC m=+3070.349822276" watchObservedRunningTime="2026-10-30 19:55:29.311012174 +0000 UTC m=+3077.346524711"
Oct 30 19:55:30.027748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:30.027714    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"]
Oct 30 19:55:30.040500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:30.040467    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4ac577-fsnn7"]
Oct 30 19:55:30.098750 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:30.098719    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3043364-7216-4a0d-83d9-ad302371c599" path="/var/lib/kubelet/pods/d3043364-7216-4a0d-83d9-ad302371c599/volumes"
Oct 30 19:55:40.628638 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.628590    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 19:55:40.634499 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.634478    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.645236 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.645209    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 19:55:40.649394 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.649373    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.653023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.653004    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf314f-openshift-default-dockercfg-sjttg\""
Oct 30 19:55:40.661212 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.661186    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 19:55:40.665664 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665640    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.665790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.665790 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665775    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.665876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665819    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhzd\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.665876 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665855    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.665944 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665882    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.665989 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.665940    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.666044 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666012    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666067    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.666143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666114    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.666184 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666141    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.666230 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666182    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.666332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666248    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666275    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666332 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666310    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666333    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666367    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxgl\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.666505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.666398    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.695615 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.688331    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 19:55:40.767297 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767261    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767304    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.767471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767346    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhzd\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767378    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767471 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767410    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767488    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767561    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.767710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767649    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767710 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767694    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767717    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767758    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767817    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767842    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767871    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.767892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767893    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.768213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767919    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxgl\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.768213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767954    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.768213 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.767983    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.768393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.768378    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.769288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.769262    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.769472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.769456    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.770012 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.769982    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.770508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770044    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.770508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770204    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.770508 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770363    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.770751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770520    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.770751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770670    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.770751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.770722    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.771158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.771138    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.772224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.772205    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.772493 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.772466    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.772758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.772740    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.787095 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.787068    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.787260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.787239    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.787515 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.787496    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhzd\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.787787 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.787767    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxgl\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:40.951486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.951459    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:40.967897 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:40.967873    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:41.276040 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.276013    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 19:55:41.277250 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:55:41.277226    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314e2376_43ec_4006_a31e_5d426ebbf9a2.slice/crio-613b5170a2e34284d1929246fe9040a0ee24a19cc43f03abe8ab46b84827c3a5 WatchSource:0}: Error finding container 613b5170a2e34284d1929246fe9040a0ee24a19cc43f03abe8ab46b84827c3a5: Status 404 returned error can't find the container with id 613b5170a2e34284d1929246fe9040a0ee24a19cc43f03abe8ab46b84827c3a5
Oct 30 19:55:41.280204 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.280165    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.280299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.280234    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.280299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.280263    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.314311 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.314287    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 19:55:41.317509 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:55:41.317486    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99864e99_111a_44e4_bd64_8be3c2ec3387.slice/crio-b34ff8783d460fb02fe81423f5bb837383ab24362270d83755fe6eb7d35d8e82 WatchSource:0}: Error finding container b34ff8783d460fb02fe81423f5bb837383ab24362270d83755fe6eb7d35d8e82: Status 404 returned error can't find the container with id b34ff8783d460fb02fe81423f5bb837383ab24362270d83755fe6eb7d35d8e82
Oct 30 19:55:41.320751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.320721    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.320844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.320788    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.320844 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.320821    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:41.420513 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.420459    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" event={"ID":"99864e99-111a-44e4-bd64-8be3c2ec3387","Type":"ContainerStarted","Data":"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"}
Oct 30 19:55:41.420741 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.420553    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" event={"ID":"99864e99-111a-44e4-bd64-8be3c2ec3387","Type":"ContainerStarted","Data":"b34ff8783d460fb02fe81423f5bb837383ab24362270d83755fe6eb7d35d8e82"}
Oct 30 19:55:41.423869 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.423834    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" event={"ID":"314e2376-43ec-4006-a31e-5d426ebbf9a2","Type":"ContainerStarted","Data":"d668053e13e8d5303914a51dc84f7f6d784e0f43bef589f53096d59dbc8137b7"}
Oct 30 19:55:41.424016 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.423890    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" event={"ID":"314e2376-43ec-4006-a31e-5d426ebbf9a2","Type":"ContainerStarted","Data":"613b5170a2e34284d1929246fe9040a0ee24a19cc43f03abe8ab46b84827c3a5"}
Oct 30 19:55:41.470347 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.469905    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podStartSLOduration=1.469887902 podStartE2EDuration="1.469887902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:41.464439454 +0000 UTC m=+3089.499951917" watchObservedRunningTime="2026-10-30 19:55:41.469887902 +0000 UTC m=+3089.505400372"
Oct 30 19:55:41.508528 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.508477    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podStartSLOduration=1.508463731 podStartE2EDuration="1.508463731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:41.50671887 +0000 UTC m=+3089.542231334" watchObservedRunningTime="2026-10-30 19:55:41.508463731 +0000 UTC m=+3089.543976194"
Oct 30 19:55:41.951858 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.951822    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:41.968404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:41.968376    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:42.954425 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:42.954384    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:42.954932 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:42.954449    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:42.993622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:42.992783    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:42.993622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:42.992857    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:43.951964 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:43.951930    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:43.952134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:43.951993    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:43.968477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:43.968449    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:43.968930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:43.968521    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:44.951834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:44.951786    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:44.952099 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:44.951893    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:44.968221 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:44.968187    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:44.968361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:44.968240    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:45.952330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:45.952297    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:45.952772 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:45.952358    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:45.972620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:45.968679    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:45.972620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:45.968748    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:46.951947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:46.951913    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:46.952135 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:46.951964    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:46.968273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:46.968233    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:46.968705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:46.968279    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:47.952492 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:47.952461    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:55:47.952703 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:47.952513    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 19:55:47.968825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:47.968791    4320 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:55:47.969303 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:47.968845    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:55:48.956921 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.956889    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:48.957282 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.957257    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:48.958114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.958095    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 19:55:48.974785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.974759    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:48.975211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.975043    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:48.975718 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:48.975699    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 19:55:54.942215 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:54.942137    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:55:54.942701 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:55:54.942367    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy" containerID="cri-o://a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2" gracePeriod=30
Oct 30 19:55:58.645791 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:55:58.645751    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:56:00.235379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.235353    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:56:00.319361 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319329    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319435    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319545 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319484    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319552    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319698 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319669    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319733    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.319887 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319843    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.320509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319904    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.320509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.319992    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\") pod \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\" (UID: \"390c2a2d-bc49-4e8e-b78a-52d625e9189b\") "
Oct 30 19:56:00.320509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.320469    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:00.320734 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.320713    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:00.320816 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.320728    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:00.321137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321109    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:00.321301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321273    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:00.321365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321316    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.321365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321343    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.321365 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321357    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.321530 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.321373    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.323198 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.323172    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:00.324440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.324413    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:00.324440 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.324420    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:00.325739 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.325715    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj" pod "390c2a2d-bc49-4e8e-b78a-52d625e9189b" (UID: "390c2a2d-bc49-4e8e-b78a-52d625e9189b"). InnerVolumeSpecName "kube-api-access-62cbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:00.422575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.422532    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.422575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.422567    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.422575 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.422580    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.422837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.422592    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.422837 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.422618    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62cbj\" (UniqueName: \"kubernetes.io/projected/390c2a2d-bc49-4e8e-b78a-52d625e9189b-kube-api-access-62cbj\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:00.769031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.768992    4320 generic.go:356] "Generic (PLEG): container finished" podID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerID="a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2" exitCode=0
Oct 30 19:56:00.769199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.769041    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"
Oct 30 19:56:00.769199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.769079    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" event={"ID":"390c2a2d-bc49-4e8e-b78a-52d625e9189b","Type":"ContainerDied","Data":"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"}
Oct 30 19:56:00.769199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.769136    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd" event={"ID":"390c2a2d-bc49-4e8e-b78a-52d625e9189b","Type":"ContainerDied","Data":"cd56edfab23083ae437f19fa9c891750e7208e7b52edabd830076c78978c73ed"}
Oct 30 19:56:00.769199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.769166    4320 scope.go:122] "RemoveContainer" containerID="a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"
Oct 30 19:56:00.781147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.781127    4320 scope.go:122] "RemoveContainer" containerID="a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"
Oct 30 19:56:00.781458 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:00.781432    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2\": container with ID starting with a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2 not found: ID does not exist" containerID="a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"
Oct 30 19:56:00.781563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.781466    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2"} err="failed to get container status \"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2\": rpc error: code = NotFound desc = could not find container \"a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2\": container with ID starting with a211e91fbb089d1ac5e895dfa9805fb76942a5a0b30087c5e53c6a93f9cd83c2 not found: ID does not exist"
Oct 30 19:56:00.825472 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.825442    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:56:00.836562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:00.836537    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-7lcrd"]
Oct 30 19:56:02.097296 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:02.097262    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" path="/var/lib/kubelet/pods/390c2a2d-bc49-4e8e-b78a-52d625e9189b/volumes"
Oct 30 19:56:14.238505 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:14.238473    4320 scope.go:122] "RemoveContainer" containerID="993374971a01f2f93284275d0d634a9b96b17b17a0e1cf52f71a3cab28ad4091"
Oct 30 19:56:14.248288 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:14.248261    4320 scope.go:122] "RemoveContainer" containerID="8c4816167f578d5d4da637e9844dd02dffdeb0ce92133cdbbd897bffe5231333"
Oct 30 19:56:14.260631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:14.260584    4320 scope.go:122] "RemoveContainer" containerID="69ac61a8c0c08abd69473922b506dc7ebefdac730720c8c297725d993e3e71cb"
Oct 30 19:56:20.092771 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:20.092723    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:56:28.644552 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:28.644503    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:56:37.067376 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:37.067343    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:56:37.098900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:37.098863    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:56:41.091374 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.091341    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:56:41.091971 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.091505    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" podUID="51278123-0a2f-419d-8204-5065b601c10b" containerName="maas-api" containerID="cri-o://4c1c5b165fd4a6fe9d4104724db14c9dd5119d7fd8fdfb927483b6698f31ab77" gracePeriod=30
Oct 30 19:56:41.806678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.806638    4320 generic.go:356] "Generic (PLEG): container finished" podID="51278123-0a2f-419d-8204-5065b601c10b" containerID="4c1c5b165fd4a6fe9d4104724db14c9dd5119d7fd8fdfb927483b6698f31ab77" exitCode=0
Oct 30 19:56:41.806872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.806731    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" event={"ID":"51278123-0a2f-419d-8204-5065b601c10b","Type":"ContainerDied","Data":"4c1c5b165fd4a6fe9d4104724db14c9dd5119d7fd8fdfb927483b6698f31ab77"}
Oct 30 19:56:41.806872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.806780    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k" event={"ID":"51278123-0a2f-419d-8204-5065b601c10b","Type":"ContainerDied","Data":"eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7"}
Oct 30 19:56:41.806872 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.806803    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf7325cc137e3f625ced996b7bbca5d84967ac131efc077d1ae156156e9f3b7"
Oct 30 19:56:41.810841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.810812    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:56:41.936321 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.936279    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\") pod \"51278123-0a2f-419d-8204-5065b601c10b\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") "
Oct 30 19:56:41.936506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.936353    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\") pod \"51278123-0a2f-419d-8204-5065b601c10b\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") "
Oct 30 19:56:41.936506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.936462    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\") pod \"51278123-0a2f-419d-8204-5065b601c10b\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") "
Oct 30 19:56:41.936755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.936715    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca" pod "51278123-0a2f-419d-8204-5065b601c10b" (UID: "51278123-0a2f-419d-8204-5065b601c10b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:41.936927 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.936726    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\" (UniqueName: \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\") pod \"51278123-0a2f-419d-8204-5065b601c10b\" (UID: \"51278123-0a2f-419d-8204-5065b601c10b\") "
Oct 30 19:56:41.937653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.937629    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51278123-0a2f-419d-8204-5065b601c10b-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:41.939919 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.939884    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls" pod "51278123-0a2f-419d-8204-5065b601c10b" (UID: "51278123-0a2f-419d-8204-5065b601c10b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:41.940588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.940562    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs" pod "51278123-0a2f-419d-8204-5065b601c10b" (UID: "51278123-0a2f-419d-8204-5065b601c10b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:41.940588 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:41.940588    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6" pod "51278123-0a2f-419d-8204-5065b601c10b" (UID: "51278123-0a2f-419d-8204-5065b601c10b"). InnerVolumeSpecName "kube-api-access-46mn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:42.039160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.039104    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:42.039160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.039129    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51278123-0a2f-419d-8204-5065b601c10b-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:42.039160 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.039140    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mn6\" (UniqueName: \"kubernetes.io/projected/51278123-0a2f-419d-8204-5065b601c10b-kube-api-access-46mn6\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:42.814574 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.814536    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"
Oct 30 19:56:42.864563 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.864529    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:56:42.867987 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.867960    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:56:42.875650 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.875621    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-5250ee-6c6fddc59f-c985k"]
Oct 30 19:56:42.906143 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:42.906115    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:56:44.102451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:44.102408    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51278123-0a2f-419d-8204-5065b601c10b" path="/var/lib/kubelet/pods/51278123-0a2f-419d-8204-5065b601c10b/volumes"
Oct 30 19:56:45.736266 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.736228    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:45.736823 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.736302    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:45.838494 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.838465    4320 generic.go:356] "Generic (PLEG): container finished" podID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerID="bfb769ea5d07bf565958d71d249c4417ad2b6303290646d371595dc6a0d7c537" exitCode=0
Oct 30 19:56:45.838697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.838539    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerDied","Data":"bfb769ea5d07bf565958d71d249c4417ad2b6303290646d371595dc6a0d7c537"}
Oct 30 19:56:45.839057 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.839040    4320 scope.go:122] "RemoveContainer" containerID="bfb769ea5d07bf565958d71d249c4417ad2b6303290646d371595dc6a0d7c537"
Oct 30 19:56:45.840796 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.840762    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:45.840866 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.840837    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:45.840907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:45.840881    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:46.848572 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:46.848537    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerStarted","Data":"7be60f5d89534b4be827575dac500ea5cdbf8adeed12a35072d0ec25645900aa"}
Oct 30 19:56:46.993849 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:46.993809    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:56:46.994199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:46.994176    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:46.994277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:46.994224    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:47.876621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:47.876144    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing" containerID="cri-o://fc4d48547d520edd325c0c90e3e3c7275be01b43712858fc4b85c039685ba6de" gracePeriod=25
Oct 30 19:56:47.916623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:47.907663    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing" containerID="cri-o://5ecfc22d398db1061b5f6c7697bb7ae69f904ba1831a5d23c8fb8233704db07f" gracePeriod=25
Oct 30 19:56:47.997621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:47.997259    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:47.997621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:47.997328    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:48.993947 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:48.993912    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:48.994371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:48.993978    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:49.368685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.366034    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:56:49.368685 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.366274    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy" containerID="cri-o://d3af5d873325b94346ba29a7ae049832b31f33ceb15016f4453b495c04e0a848" gracePeriod=30
Oct 30 19:56:49.798522 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.798483    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"]
Oct 30 19:56:49.799227 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799210    4320 cpu_manager.go:414] "containerMap: removing container" podUID="51278123-0a2f-419d-8204-5065b601c10b" containerName="maas-api"
Oct 30 19:56:49.799316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799229    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51278123-0a2f-419d-8204-5065b601c10b" containerName="maas-api"
Oct 30 19:56:49.799316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799248    4320 cpu_manager.go:414] "containerMap: removing container" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy"
Oct 30 19:56:49.799316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799254    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy"
Oct 30 19:56:49.799474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799409    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="51278123-0a2f-419d-8204-5065b601c10b" containerName="maas-api"
Oct 30 19:56:49.799474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.799420    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="390c2a2d-bc49-4e8e-b78a-52d625e9189b" containerName="istio-proxy"
Oct 30 19:56:49.802238 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.802222    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:49.818379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.818356    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:49.818657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.818633    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:56:49.823232 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.823206    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"]
Oct 30 19:56:49.916147 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.916113    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:49.916310 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.916267    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:49.916406 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.916389    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8mw\" (UniqueName: \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:49.995841 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.995806    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:49.996462 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:49.996388    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:50.000387 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.000362    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 19:56:50.007920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.006924    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 19:56:50.007920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.007036    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.023553 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.023381    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5a0a17-openshift-default-dockercfg-xfwqt\""
Oct 30 19:56:50.024299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.024274    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.025083 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.025059    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.025276 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.025257    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.025430 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.025409    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8mw\" (UniqueName: \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.027943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.027916    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.044315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.044293    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8mw\" (UniqueName: \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.115335 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.115266    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:50.126961 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.126931    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127067 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127051    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127200 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127178    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127279 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127256    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127413 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127324    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127480 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127423    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127592 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127556    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127732 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127712    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwmg\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.127798 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.127767    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229400 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229370    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229459    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwmg\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229498    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229566 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229561    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229620    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229757 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229706    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229771    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229832    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.229922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.229867    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.230154 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.230130    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.230233 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.230175    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.230291 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.230231    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.230474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.230451    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.230785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.230744    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.232214 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.232189    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.232488 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.232468    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.249173 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.249148    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwmg\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.249589 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.249572    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.328295 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.328262    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:50.441343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.440751    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"]
Oct 30 19:56:50.656292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.656236    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 19:56:50.699758 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.699722    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:50.699906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.699791    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:50.699906 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.699820    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:56:50.915974 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.915893    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4" event={"ID":"b9f0c11a-bc03-453e-b317-84b4148155ac","Type":"ContainerStarted","Data":"0dcf7d79a9d2bc0705a69061161fca8cbdd678421e6b9781cfa66d122ca7fa56"}
Oct 30 19:56:50.917740 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.917712    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" event={"ID":"132e6434-8b27-4a54-a4dd-f181578771cf","Type":"ContainerStarted","Data":"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"}
Oct 30 19:56:50.917928 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.917912    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" event={"ID":"132e6434-8b27-4a54-a4dd-f181578771cf","Type":"ContainerStarted","Data":"d4691c89e655d4526dd65a3eef4a7f41b8adfa8d023fae2943be62b1947f8664"}
Oct 30 19:56:50.964003 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:50.963941    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podStartSLOduration=1.963922283 podStartE2EDuration="1.963922283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:50.960683885 +0000 UTC m=+3158.996196350" watchObservedRunningTime="2026-10-30 19:56:50.963922283 +0000 UTC m=+3158.999434748"
Oct 30 19:56:51.000062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.000028    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:51.000490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.000098    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:51.330687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.329278    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:51.332186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.331788    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:51.332186 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.331864    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:51.934791 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.934751    4320 generic.go:356] "Generic (PLEG): container finished" podID="b9f0c11a-bc03-453e-b317-84b4148155ac" containerID="dde9942a5771d989749f67a667aa9cbe2754aacc05604da9d0d29577606dc212" exitCode=0
Oct 30 19:56:51.935093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.935065    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4" event={"ID":"b9f0c11a-bc03-453e-b317-84b4148155ac","Type":"ContainerDied","Data":"dde9942a5771d989749f67a667aa9cbe2754aacc05604da9d0d29577606dc212"}
Oct 30 19:56:51.993676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.993561    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:56:51.993676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:51.993642    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:56:52.124660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.124138    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" containerID="cri-o://2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb" gracePeriod=45
Oct 30 19:56:52.124660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.124323    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main" containerID="cri-o://9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff" gracePeriod=45
Oct 30 19:56:52.223062 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.219069    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 19:56:52.334683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.331724    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:52.334683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.331786    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:52.613068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.612986    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:56:52.616754 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.616717    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:56:52.713161 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.713123    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.713923 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.713897    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.714446 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.714425    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\" (UniqueName: \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.714735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.714717    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.714920 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.714900    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.715210 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.715114    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.715442 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.715423    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.715216    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716260    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716331    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\" (UniqueName: \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716363    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\") pod \"3649f898-395d-48f0-9b03-f2fd1cd021d9\" (UID: \"3649f898-395d-48f0-9b03-f2fd1cd021d9\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716392    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716590    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.716941    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\") pod \"8d690fa9-c8ce-4d82-a5be-fc7456679906\" (UID: \"8d690fa9-c8ce-4d82-a5be-fc7456679906\") "
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.717749    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.717772    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.717971    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.718143    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.718317    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.718379    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.718450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.718427    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.719192 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.718464    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.719487 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.719466    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "kube-api-access-nk57s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.721938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.721915    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "kube-api-access-znpbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.722433 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.722413    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs" pod "8d690fa9-c8ce-4d82-a5be-fc7456679906" (UID: "8d690fa9-c8ce-4d82-a5be-fc7456679906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:52.724548 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.724513    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs" pod "3649f898-395d-48f0-9b03-f2fd1cd021d9" (UID: "3649f898-395d-48f0-9b03-f2fd1cd021d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:52.819401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819369    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3649f898-395d-48f0-9b03-f2fd1cd021d9-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819398    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819401 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819407    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk57s\" (UniqueName: \"kubernetes.io/projected/8d690fa9-c8ce-4d82-a5be-fc7456679906-kube-api-access-nk57s\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819417    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819427    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819444    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819452    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3649f898-395d-48f0-9b03-f2fd1cd021d9-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819459    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d690fa9-c8ce-4d82-a5be-fc7456679906-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819466    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znpbd\" (UniqueName: \"kubernetes.io/projected/3649f898-395d-48f0-9b03-f2fd1cd021d9-kube-api-access-znpbd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.819670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.819474    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d690fa9-c8ce-4d82-a5be-fc7456679906-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:52.944652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.944591    4320 generic.go:356] "Generic (PLEG): container finished" podID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerID="2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb" exitCode=0
Oct 30 19:56:52.944652 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.944649    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" event={"ID":"3649f898-395d-48f0-9b03-f2fd1cd021d9","Type":"ContainerDied","Data":"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"}
Oct 30 19:56:52.944911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.944692    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h" event={"ID":"3649f898-395d-48f0-9b03-f2fd1cd021d9","Type":"ContainerDied","Data":"bcfdbbf7ba51af7ee3637c7ecc5bb7c2904f127ab1e0588b8ec87c2dc55d3a68"}
Oct 30 19:56:52.944911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.944715    4320 scope.go:122] "RemoveContainer" containerID="2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"
Oct 30 19:56:52.944911 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.944655    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"
Oct 30 19:56:52.946697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.946670    4320 generic.go:356] "Generic (PLEG): container finished" podID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerID="9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff" exitCode=0
Oct 30 19:56:52.946834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.946715    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"
Oct 30 19:56:52.946834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.946757    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" event={"ID":"8d690fa9-c8ce-4d82-a5be-fc7456679906","Type":"ContainerDied","Data":"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"}
Oct 30 19:56:52.946834 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.946794    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j" event={"ID":"8d690fa9-c8ce-4d82-a5be-fc7456679906","Type":"ContainerDied","Data":"d6b5203f579769fd62bcb69ec254b9de69a0f9356a04b9ac44104bb8c6b15095"}
Oct 30 19:56:52.958007 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.957987    4320 scope.go:122] "RemoveContainer" containerID="2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"
Oct 30 19:56:52.958696 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:52.958668    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb\": container with ID starting with 2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb not found: ID does not exist" containerID="2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"
Oct 30 19:56:52.958922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.958703    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb"} err="failed to get container status \"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb\": rpc error: code = NotFound desc = could not find container \"2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb\": container with ID starting with 2e04f06540a939f6a2c0034be1d110d8175acfad1462ebba8182feaaa3cf93bb not found: ID does not exist"
Oct 30 19:56:52.958922 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.958727    4320 scope.go:122] "RemoveContainer" containerID="9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"
Oct 30 19:56:52.969969 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.969889    4320 scope.go:122] "RemoveContainer" containerID="9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"
Oct 30 19:56:52.974014 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:52.973941    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff\": container with ID starting with 9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff not found: ID does not exist" containerID="9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"
Oct 30 19:56:52.974014 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:52.973975    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff"} err="failed to get container status \"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff\": rpc error: code = NotFound desc = could not find container \"9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff\": container with ID starting with 9c6985b553f1031bbe81f57f6b255cbe11bd133320b4340c1c476e917803fbff not found: ID does not exist"
Oct 30 19:56:53.004587 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.004009    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:56:53.006404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.005539    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:56:53.006404 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.005864    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:56:53.006907 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.006868    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:56:53.008063 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.008040    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/premium-w3-249dd4-kserve-698b458758-lsg6h"]
Oct 30 19:56:53.047840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.047776    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:56:53.052202 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.052174    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-249dd4/simulator-w3-249dd4-kserve-b47dbc8dc-brh2j"]
Oct 30 19:56:53.170683 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.169404    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:53.265038 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.264941    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\" (UniqueName: \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\") pod \"b9f0c11a-bc03-453e-b317-84b4148155ac\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") "
Oct 30 19:56:53.267735 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.267706    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\") pod \"b9f0c11a-bc03-453e-b317-84b4148155ac\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") "
Oct 30 19:56:53.268093 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.268071    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\") pod \"b9f0c11a-bc03-453e-b317-84b4148155ac\" (UID: \"b9f0c11a-bc03-453e-b317-84b4148155ac\") "
Oct 30 19:56:53.268466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.268443    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca" pod "b9f0c11a-bc03-453e-b317-84b4148155ac" (UID: "b9f0c11a-bc03-453e-b317-84b4148155ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:53.274634 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.268763    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw" pod "b9f0c11a-bc03-453e-b317-84b4148155ac" (UID: "b9f0c11a-bc03-453e-b317-84b4148155ac"). InnerVolumeSpecName "kube-api-access-df8mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:53.296395 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.296370    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:53.296562 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.296545    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8mw\" (UniqueName: \"kubernetes.io/projected/b9f0c11a-bc03-453e-b317-84b4148155ac-kube-api-access-df8mw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:53.323953 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.319063    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca" pod "b9f0c11a-bc03-453e-b317-84b4148155ac" (UID: "b9f0c11a-bc03-453e-b317-84b4148155ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:53.329621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.329575    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:53.329751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.329711    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:53.398649 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.398589    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f0c11a-bc03-453e-b317-84b4148155ac-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:53.957469 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.957437    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"
Oct 30 19:56:53.957720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.957468    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4" event={"ID":"b9f0c11a-bc03-453e-b317-84b4148155ac","Type":"ContainerDied","Data":"0dcf7d79a9d2bc0705a69061161fca8cbdd678421e6b9781cfa66d122ca7fa56"}
Oct 30 19:56:53.957720 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:53.957519    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dcf7d79a9d2bc0705a69061161fca8cbdd678421e6b9781cfa66d122ca7fa56"
Oct 30 19:56:54.045448 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:54.045410    4320 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f0c11a_bc03_453e_b317_84b4148155ac.slice/crio-0dcf7d79a9d2bc0705a69061161fca8cbdd678421e6b9781cfa66d122ca7fa56\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f0c11a_bc03_453e_b317_84b4148155ac.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:56:54.105900 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.105867    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" path="/var/lib/kubelet/pods/3649f898-395d-48f0-9b03-f2fd1cd021d9/volumes"
Oct 30 19:56:54.106540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.106518    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" path="/var/lib/kubelet/pods/8d690fa9-c8ce-4d82-a5be-fc7456679906/volumes"
Oct 30 19:56:54.352495 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.352407    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:54.423622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.406867    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:54.984628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.976465    4320 generic.go:356] "Generic (PLEG): container finished" podID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerID="d3af5d873325b94346ba29a7ae049832b31f33ceb15016f4453b495c04e0a848" exitCode=0
Oct 30 19:56:54.984628 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:54.977486    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" event={"ID":"0cc5bed6-279f-4180-b9de-9403b9b5f1b9","Type":"ContainerDied","Data":"d3af5d873325b94346ba29a7ae049832b31f33ceb15016f4453b495c04e0a848"}
Oct 30 19:56:55.135620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.134423    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:56:55.169125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.169092    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.175804 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.175774    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.176041 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.176022    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.176190 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.176173    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.176368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.176351    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.176486 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.176472    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.177546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.177167    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:55.178090 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.178004    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:55.178325 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.178302    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:55.179789 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.179761    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.180146 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.180121    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.180235 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.180188    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\") pod \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\" (UID: \"0cc5bed6-279f-4180-b9de-9403b9b5f1b9\") "
Oct 30 19:56:55.180670 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.180645    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:55.180991 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.180968    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:55.182289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.182254    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "kube-api-access-6wstf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184706    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184728    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wstf\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-kube-api-access-6wstf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184742    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184763    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184778    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.191708 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.184791    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.194236 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.192483    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:55.214270 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.214234    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:55.215579 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.215553    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token" pod "0cc5bed6-279f-4180-b9de-9403b9b5f1b9" (UID: "0cc5bed6-279f-4180-b9de-9403b9b5f1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:55.294788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.292895    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.294788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.292929    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.294788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.292944    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc5bed6-279f-4180-b9de-9403b9b5f1b9-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:56:55.329284 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.329254    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:55.329432 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.329309    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:55.996622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.993232    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc" event={"ID":"0cc5bed6-279f-4180-b9de-9403b9b5f1b9","Type":"ContainerDied","Data":"da179d3f5a74cb2662e90b5f79880d215e34add69ec8185c8f6b0b9ddfa89877"}
Oct 30 19:56:55.996622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.993311    4320 scope.go:122] "RemoveContainer" containerID="d3af5d873325b94346ba29a7ae049832b31f33ceb15016f4453b495c04e0a848"
Oct 30 19:56:55.996622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:55.993449    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"
Oct 30 19:56:56.047451 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.047421    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:56:56.059224 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.059196    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-vxglc"]
Oct 30 19:56:56.105490 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.105452    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" path="/var/lib/kubelet/pods/0cc5bed6-279f-4180-b9de-9403b9b5f1b9/volumes"
Oct 30 19:56:56.328950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.328858    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:56:56.328950 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.328913    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 19:56:56.812251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.812204    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:56.812639 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:56.812614    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:57.334152 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:57.334122    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:57.334653 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:57.334631    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:57.335330 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:57.335313    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 19:56:58.645638 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:56:58.645588    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:56:58.834162 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:58.834127    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:56:58.834371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:56:58.834349    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy" containerID="cri-o://3b5a6097c85ead176d2f93cab0cd9a9e91af2b9e4498d7211fc7a2def01a480e" gracePeriod=30
Oct 30 19:57:04.062655 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.062573    4320 generic.go:356] "Generic (PLEG): container finished" podID="5715dc59-ae45-49d0-a249-de008c9765b0" containerID="3b5a6097c85ead176d2f93cab0cd9a9e91af2b9e4498d7211fc7a2def01a480e" exitCode=0
Oct 30 19:57:04.063004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.062872    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" event={"ID":"5715dc59-ae45-49d0-a249-de008c9765b0","Type":"ContainerDied","Data":"3b5a6097c85ead176d2f93cab0cd9a9e91af2b9e4498d7211fc7a2def01a480e"}
Oct 30 19:57:04.179620 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.176654    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:57:04.244110 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244081    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244135    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244195    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244300 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244226    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244326    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244361    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244462    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244515    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.244699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.244659    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\") pod \"5715dc59-ae45-49d0-a249-de008c9765b0\" (UID: \"5715dc59-ae45-49d0-a249-de008c9765b0\") "
Oct 30 19:57:04.245464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245221    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:04.245464 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245404    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:04.245661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245483    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:04.245788 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245771    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5715dc59-ae45-49d0-a249-de008c9765b0-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.245843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245793    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.245843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.245803    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.246829 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.246792    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:04.247518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.247485    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "kube-api-access-8vn98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:04.248565 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.248539    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:04.249705 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.249678    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:04.255450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.254849    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:04.256001 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.255976    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token" pod "5715dc59-ae45-49d0-a249-de008c9765b0" (UID: "5715dc59-ae45-49d0-a249-de008c9765b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351079    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vn98\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-kube-api-access-8vn98\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351114    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5715dc59-ae45-49d0-a249-de008c9765b0-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351129    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5715dc59-ae45-49d0-a249-de008c9765b0-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351142    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351155    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:04.351292 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:04.351178    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5715dc59-ae45-49d0-a249-de008c9765b0-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:05.095055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:05.095012    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7" event={"ID":"5715dc59-ae45-49d0-a249-de008c9765b0","Type":"ContainerDied","Data":"d8f3dfafdfdc1dc3c6607e9601cbf748bb4785c169136f2bcb17daff36ff9cd3"}
Oct 30 19:57:05.095552 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:05.095067    4320 scope.go:122] "RemoveContainer" containerID="3b5a6097c85ead176d2f93cab0cd9a9e91af2b9e4498d7211fc7a2def01a480e"
Oct 30 19:57:05.095552 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:05.095202    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"
Oct 30 19:57:05.161658 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:05.161629    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:57:05.164785 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:05.164764    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-95fx7"]
Oct 30 19:57:06.104157 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:06.104118    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" path="/var/lib/kubelet/pods/5715dc59-ae45-49d0-a249-de008c9765b0/volumes"
Oct 30 19:57:06.797621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:06.796916    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:57:06.804619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:06.799744    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:57:08.156756 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.156567    4320 generic.go:356] "Generic (PLEG): container finished" podID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerID="fc4d48547d520edd325c0c90e3e3c7275be01b43712858fc4b85c039685ba6de" exitCode=0
Oct 30 19:57:08.157223 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.156863    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" event={"ID":"f40c2549-5f26-4370-b4fe-d214a17fb445","Type":"ContainerDied","Data":"fc4d48547d520edd325c0c90e3e3c7275be01b43712858fc4b85c039685ba6de"}
Oct 30 19:57:08.159142 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.159117    4320 generic.go:356] "Generic (PLEG): container finished" podID="861d5d2b-e377-4710-9f93-f2e79194440a" containerID="5ecfc22d398db1061b5f6c7697bb7ae69f904ba1831a5d23c8fb8233704db07f" exitCode=0
Oct 30 19:57:08.159301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.159157    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" event={"ID":"861d5d2b-e377-4710-9f93-f2e79194440a","Type":"ContainerDied","Data":"5ecfc22d398db1061b5f6c7697bb7ae69f904ba1831a5d23c8fb8233704db07f"}
Oct 30 19:57:08.302055 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.301875    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:57:08.308518 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.308497    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:57:08.416379 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.416298    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:57:08.417064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417043    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy"
Oct 30 19:57:08.417064 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417065    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417079    4320 cpu_manager.go:414] "containerMap: removing container" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417086    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417124    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b9f0c11a-bc03-453e-b317-84b4148155ac" containerName="revoke-keys"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417132    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f0c11a-bc03-453e-b317-84b4148155ac" containerName="revoke-keys"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417183    4320 cpu_manager.go:414] "containerMap: removing container" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417191    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417204    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417212    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417222    4320 cpu_manager.go:414] "containerMap: removing container" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy"
Oct 30 19:57:08.417237 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417228    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417244    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417251    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417414    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5715dc59-ae45-49d0-a249-de008c9765b0" containerName="istio-proxy"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417429    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="3649f898-395d-48f0-9b03-f2fd1cd021d9" containerName="main"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417447    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f0c11a-bc03-453e-b317-84b4148155ac" containerName="revoke-keys"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417464    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" containerName="payload-pre-processing"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417473    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d690fa9-c8ce-4d82-a5be-fc7456679906" containerName="main"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417482    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc5bed6-279f-4180-b9de-9403b9b5f1b9" containerName="istio-proxy"
Oct 30 19:57:08.417768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.417494    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" containerName="payload-processing"
Oct 30 19:57:08.420569 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.420551    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.442124 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.437688    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e6016\""
Oct 30 19:57:08.444990 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.444949    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.469283    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\") pod \"861d5d2b-e377-4710-9f93-f2e79194440a\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") "
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.469338    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\" (UniqueName: \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\") pod \"f40c2549-5f26-4370-b4fe-d214a17fb445\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") "
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.469500    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\" (UniqueName: \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\") pod \"861d5d2b-e377-4710-9f93-f2e79194440a\" (UID: \"861d5d2b-e377-4710-9f93-f2e79194440a\") "
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.469542    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\") pod \"f40c2549-5f26-4370-b4fe-d214a17fb445\" (UID: \"f40c2549-5f26-4370-b4fe-d214a17fb445\") "
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.505489    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume" pod "f40c2549-5f26-4370-b4fe-d214a17fb445" (UID: "f40c2549-5f26-4370-b4fe-d214a17fb445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.506194    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40c2549-5f26-4370-b4fe-d214a17fb445-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.517000    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume" pod "861d5d2b-e377-4710-9f93-f2e79194440a" (UID: "861d5d2b-e377-4710-9f93-f2e79194440a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:08.543301 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.534805    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl" pod "861d5d2b-e377-4710-9f93-f2e79194440a" (UID: "861d5d2b-e377-4710-9f93-f2e79194440a"). InnerVolumeSpecName "kube-api-access-svlrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:08.546043 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.546001    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e6016-dockercfg-r9cqg\""
Oct 30 19:57:08.644861 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.630677    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.645343 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.645306    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9r5l\" (UniqueName: \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.645722 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.645694    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svlrl\" (UniqueName: \"kubernetes.io/projected/861d5d2b-e377-4710-9f93-f2e79194440a-kube-api-access-svlrl\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:08.645873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.645859    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/861d5d2b-e377-4710-9f93-f2e79194440a-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:08.675960 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.675849    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48" pod "f40c2549-5f26-4370-b4fe-d214a17fb445" (UID: "f40c2549-5f26-4370-b4fe-d214a17fb445"). InnerVolumeSpecName "kube-api-access-fzs48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:08.747069 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.747032    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.747256 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.747174    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9r5l\" (UniqueName: \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.747316 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.747277    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzs48\" (UniqueName: \"kubernetes.io/projected/f40c2549-5f26-4370-b4fe-d214a17fb445-kube-api-access-fzs48\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:08.747854 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.747833    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:08.766114 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:08.766090    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9r5l\" (UniqueName: \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:09.048678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.046746    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:09.169031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.168995    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"
Oct 30 19:57:09.169525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.169004    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7" event={"ID":"861d5d2b-e377-4710-9f93-f2e79194440a","Type":"ContainerDied","Data":"0064ea6b9be77feef2d1f345f6aab053ef867bea32d7847babc68a8f45cd4933"}
Oct 30 19:57:09.169525 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.169156    4320 scope.go:122] "RemoveContainer" containerID="5ecfc22d398db1061b5f6c7697bb7ae69f904ba1831a5d23c8fb8233704db07f"
Oct 30 19:57:09.170893 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.170829    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56" event={"ID":"f40c2549-5f26-4370-b4fe-d214a17fb445","Type":"ContainerDied","Data":"4fb65ac82c3eea6c15ef63fc2b7666f881eb9e5b4d4fef19c4aec6f928b71dac"}
Oct 30 19:57:09.170949 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.170933    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"
Oct 30 19:57:09.181561 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.181537    4320 scope.go:122] "RemoveContainer" containerID="fc4d48547d520edd325c0c90e3e3c7275be01b43712858fc4b85c039685ba6de"
Oct 30 19:57:09.234997 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.234974    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:57:09.238519 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.238499    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-249dd4-86c754dc87-7pzn7"]
Oct 30 19:57:09.271368 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.271344    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:57:09.274393 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.274369    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-249dd4-548548cc5d-rxs56"]
Oct 30 19:57:09.359546 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:57:09.359521    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d153b5e_947c_42ea_b996_b8582bb84528.slice/crio-6b9d9e63a41bc101dc5d97b2df0b7723eebd3f6333cf5448f3f3f059fa5661c3 WatchSource:0}: Error finding container 6b9d9e63a41bc101dc5d97b2df0b7723eebd3f6333cf5448f3f3f059fa5661c3: Status 404 returned error can't find the container with id 6b9d9e63a41bc101dc5d97b2df0b7723eebd3f6333cf5448f3f3f059fa5661c3
Oct 30 19:57:09.359687 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:09.359667    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:57:10.096856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.096826    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861d5d2b-e377-4710-9f93-f2e79194440a" path="/var/lib/kubelet/pods/861d5d2b-e377-4710-9f93-f2e79194440a/volumes"
Oct 30 19:57:10.097251 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.097237    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40c2549-5f26-4370-b4fe-d214a17fb445" path="/var/lib/kubelet/pods/f40c2549-5f26-4370-b4fe-d214a17fb445/volumes"
Oct 30 19:57:10.179356 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.179321    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" event={"ID":"0d153b5e-947c-42ea-b996-b8582bb84528","Type":"ContainerStarted","Data":"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"}
Oct 30 19:57:10.179800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.179372    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" event={"ID":"0d153b5e-947c-42ea-b996-b8582bb84528","Type":"ContainerStarted","Data":"6b9d9e63a41bc101dc5d97b2df0b7723eebd3f6333cf5448f3f3f059fa5661c3"}
Oct 30 19:57:10.179800 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.179399    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:10.222222 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:10.222176    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" podStartSLOduration=2.222160485 podStartE2EDuration="2.222160485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:10.217142026 +0000 UTC m=+3178.252654490" watchObservedRunningTime="2026-10-30 19:57:10.222160485 +0000 UTC m=+3178.257672952"
Oct 30 19:57:11.190005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.189977    4320 generic.go:356] "Generic (PLEG): container finished" podID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerID="8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec" exitCode=0
Oct 30 19:57:11.190503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.190040    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerDied","Data":"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"}
Oct 30 19:57:11.190503 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.190494    4320 scope.go:122] "RemoveContainer" containerID="8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"
Oct 30 19:57:11.192269 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.192240    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:57:11.192360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.192312    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:57:11.192360 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:11.192344    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:57:12.203338 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:12.203282    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerStarted","Data":"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"}
Oct 30 19:57:12.563671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:12.557211    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:57:12.930260 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:12.930226    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:57:12.930444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:12.930374    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:57:12.930444 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:12.930420    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:57:13.932621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:13.930166    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:57:13.932621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:13.930225    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:57:14.304980 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:14.304902    4320 scope.go:122] "RemoveContainer" containerID="4c1c5b165fd4a6fe9d4104724db14c9dd5119d7fd8fdfb927483b6698f31ab77"
Oct 30 19:57:14.316134 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:14.316114    4320 scope.go:122] "RemoveContainer" containerID="887e95c73e4ba68c8573a128c0c02de9e5d6cc37ab52afd754ab3dcf6faaa8a0"
Oct 30 19:57:14.325242 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:14.325221    4320 scope.go:122] "RemoveContainer" containerID="1c5a15f8fa368195b8e3e03d317cc5b6309772b1c97b9918e12977edbfea7eca"
Oct 30 19:57:14.930825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:14.930787    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:57:14.931011 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:14.930854    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:57:15.930005 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:15.929971    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:57:15.930433 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:15.930023    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:57:16.930660 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:16.930589    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:57:16.931046 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:16.930681    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:57:17.977540 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:17.977506    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:57:17.978080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:17.977991    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:57:17.979321 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:17.979292    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 19:57:19.667700 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.667668    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"]
Oct 30 19:57:19.671017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.670993    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.687507 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.687481    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"]
Oct 30 19:57:19.691467 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.691442    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.695762 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.695744    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:57:19.696661 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.696022    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:19.700955 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.700927    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"]
Oct 30 19:57:19.710466 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.710442    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"]
Oct 30 19:57:19.795485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795442    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2r7p\" (UniqueName: \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.795485 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795483    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.795736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795590    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.795736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795651    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.795943 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795914    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.796072 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.795999    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdn9\" (UniqueName: \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.897306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897272    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.897476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897315    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdn9\" (UniqueName: \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.897476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897392    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2r7p\" (UniqueName: \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.897476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897418    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.897476 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897468    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.897752 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.897635    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.898225 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.898190    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.898345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.898253    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.899761 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.899741    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.899938 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.899918    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.915682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.915654    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2r7p\" (UniqueName: \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\") pod \"maas-api-revoke-sub-c5418cce-54v58\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:19.915766 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.915745    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdn9\" (UniqueName: \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\") pod \"maas-api-revoke-sub-dd6fd5ef-b42dw\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:19.985208 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:19.985153    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:20.009903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:20.009868    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:20.298770 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:20.298741    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"]
Oct 30 19:57:20.300847 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:57:20.300810    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fa1235_87ce_4ed4_b754_070eaf825881.slice/crio-20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb WatchSource:0}: Error finding container 20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb: Status 404 returned error can't find the container with id 20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb
Oct 30 19:57:20.342848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:20.342816    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"]
Oct 30 19:57:20.343246 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:57:20.343220    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fd2164_2685_43de_b9f6_e304122c6dda.slice/crio-457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd WatchSource:0}: Error finding container 457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd: Status 404 returned error can't find the container with id 457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd
Oct 30 19:57:21.193278 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.193251    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:57:21.279523 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.279486    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw" event={"ID":"b3fd2164-2685-43de-b9f6-e304122c6dda","Type":"ContainerStarted","Data":"5082abce7d728f1239a51fbe464d064fff6ab18ef5fd792b35e101da97c5a007"}
Oct 30 19:57:21.280509 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.279556    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw" event={"ID":"b3fd2164-2685-43de-b9f6-e304122c6dda","Type":"ContainerStarted","Data":"457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd"}
Oct 30 19:57:21.282783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.282742    4320 generic.go:356] "Generic (PLEG): container finished" podID="b9fa1235-87ce-4ed4-b754-070eaf825881" containerID="d2d762f5ecd7faf173d7ef3e2addfcd98513fe5c48e76e000792d789ca66d114" exitCode=0
Oct 30 19:57:21.282903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.282811    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58" event={"ID":"b9fa1235-87ce-4ed4-b754-070eaf825881","Type":"ContainerDied","Data":"d2d762f5ecd7faf173d7ef3e2addfcd98513fe5c48e76e000792d789ca66d114"}
Oct 30 19:57:21.282903 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.282850    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58" event={"ID":"b9fa1235-87ce-4ed4-b754-070eaf825881","Type":"ContainerStarted","Data":"20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb"}
Oct 30 19:57:21.322847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:21.322807    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw" podStartSLOduration=2.322793134 podStartE2EDuration="2.322793134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:21.31847037 +0000 UTC m=+3189.353982835" watchObservedRunningTime="2026-10-30 19:57:21.322793134 +0000 UTC m=+3189.358305597"
Oct 30 19:57:22.291564 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.291533    4320 generic.go:356] "Generic (PLEG): container finished" podID="b3fd2164-2685-43de-b9f6-e304122c6dda" containerID="5082abce7d728f1239a51fbe464d064fff6ab18ef5fd792b35e101da97c5a007" exitCode=0
Oct 30 19:57:22.292017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.291625    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw" event={"ID":"b3fd2164-2685-43de-b9f6-e304122c6dda","Type":"ContainerDied","Data":"5082abce7d728f1239a51fbe464d064fff6ab18ef5fd792b35e101da97c5a007"}
Oct 30 19:57:22.456855 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.456826    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:22.543319 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.543251    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\") pod \"b9fa1235-87ce-4ed4-b754-070eaf825881\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") "
Oct 30 19:57:22.543454 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.543447    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\" (UniqueName: \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\") pod \"b9fa1235-87ce-4ed4-b754-070eaf825881\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") "
Oct 30 19:57:22.543534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.543521    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\") pod \"b9fa1235-87ce-4ed4-b754-070eaf825881\" (UID: \"b9fa1235-87ce-4ed4-b754-070eaf825881\") "
Oct 30 19:57:22.543608 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.543572    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca" pod "b9fa1235-87ce-4ed4-b754-070eaf825881" (UID: "b9fa1235-87ce-4ed4-b754-070eaf825881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:22.544004 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.543986    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:22.545645 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.545580    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca" pod "b9fa1235-87ce-4ed4-b754-070eaf825881" (UID: "b9fa1235-87ce-4ed4-b754-070eaf825881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:22.545748 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.545720    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p" pod "b9fa1235-87ce-4ed4-b754-070eaf825881" (UID: "b9fa1235-87ce-4ed4-b754-070eaf825881"). InnerVolumeSpecName "kube-api-access-z2r7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:22.644954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.644925    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2r7p\" (UniqueName: \"kubernetes.io/projected/b9fa1235-87ce-4ed4-b754-070eaf825881-kube-api-access-z2r7p\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:22.644954 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:22.644952    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fa1235-87ce-4ed4-b754-070eaf825881-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:23.299585 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.299556    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"
Oct 30 19:57:23.300026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.299592    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58" event={"ID":"b9fa1235-87ce-4ed4-b754-070eaf825881","Type":"ContainerDied","Data":"20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb"}
Oct 30 19:57:23.300026 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.299653    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20f33e0d9903089dfbab21a8a026567f1e7b25e6bd0a010f7125afb55b542aeb"
Oct 30 19:57:23.465632 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.465606    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:23.556306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.556239    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\") pod \"b3fd2164-2685-43de-b9f6-e304122c6dda\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") "
Oct 30 19:57:23.556306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.556283    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\") pod \"b3fd2164-2685-43de-b9f6-e304122c6dda\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") "
Oct 30 19:57:23.556526 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.556312    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\" (UniqueName: \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\") pod \"b3fd2164-2685-43de-b9f6-e304122c6dda\" (UID: \"b3fd2164-2685-43de-b9f6-e304122c6dda\") "
Oct 30 19:57:23.556736 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.556705    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca" pod "b3fd2164-2685-43de-b9f6-e304122c6dda" (UID: "b3fd2164-2685-43de-b9f6-e304122c6dda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:23.557302 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.557274    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:23.559015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.558983    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca" pod "b3fd2164-2685-43de-b9f6-e304122c6dda" (UID: "b3fd2164-2685-43de-b9f6-e304122c6dda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:23.560107 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.560082    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9" pod "b3fd2164-2685-43de-b9f6-e304122c6dda" (UID: "b3fd2164-2685-43de-b9f6-e304122c6dda"). InnerVolumeSpecName "kube-api-access-5wdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:23.659101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.659074    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3fd2164-2685-43de-b9f6-e304122c6dda-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:23.659101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.659099    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdn9\" (UniqueName: \"kubernetes.io/projected/b3fd2164-2685-43de-b9f6-e304122c6dda-kube-api-access-5wdn9\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:57:23.834125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.834059    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 19:57:23.836023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.835988    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b3fd2164-2685-43de-b9f6-e304122c6dda" containerName="revoke-keys"
Oct 30 19:57:23.836023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.836024    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3fd2164-2685-43de-b9f6-e304122c6dda" containerName="revoke-keys"
Oct 30 19:57:23.836205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.836081    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b9fa1235-87ce-4ed4-b754-070eaf825881" containerName="revoke-keys"
Oct 30 19:57:23.836205 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.836089    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fa1235-87ce-4ed4-b754-070eaf825881" containerName="revoke-keys"
Oct 30 19:57:23.836479 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.836461    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3fd2164-2685-43de-b9f6-e304122c6dda" containerName="revoke-keys"
Oct 30 19:57:23.836541 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.836492    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fa1235-87ce-4ed4-b754-070eaf825881" containerName="revoke-keys"
Oct 30 19:57:23.841516 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.841357    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:23.859751 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.859727    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fe3e5e-serving-cert\""
Oct 30 19:57:23.861678 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.861643    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 19:57:23.962576 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.962545    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:23.962745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.962701    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:23.962745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.962734    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:23.962986 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:23.962966    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzlv\" (UniqueName: \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.064546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.064520    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzlv\" (UniqueName: \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.064744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.064584    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.064744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.064670    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.064744 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.064690    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.065445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.065416    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.067108 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.067085    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.067211 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.067128    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.083267 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.083243    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzlv\" (UniqueName: \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\") pod \"maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.172551 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.172427    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:24.323768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.319855    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw" event={"ID":"b3fd2164-2685-43de-b9f6-e304122c6dda","Type":"ContainerDied","Data":"457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd"}
Oct 30 19:57:24.323768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.319915    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457e1650e57ff8039cb74842db9b126601e103461b5b26e792c00ee8e52bc1dd"
Oct 30 19:57:24.323768 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.319982    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"
Oct 30 19:57:24.527137 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:24.527110    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 19:57:24.527900 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:57:24.527872    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c6d48c_dfc7_4aae_adbd_fd49ab1ee687.slice/crio-538a0ea7d27547eccb32df650582622d648a063292a99abd3bb33302c46c889b WatchSource:0}: Error finding container 538a0ea7d27547eccb32df650582622d648a063292a99abd3bb33302c46c889b: Status 404 returned error can't find the container with id 538a0ea7d27547eccb32df650582622d648a063292a99abd3bb33302c46c889b
Oct 30 19:57:25.328346 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:25.328306    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" event={"ID":"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687","Type":"ContainerStarted","Data":"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"}
Oct 30 19:57:25.328840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:25.328369    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" event={"ID":"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687","Type":"ContainerStarted","Data":"538a0ea7d27547eccb32df650582622d648a063292a99abd3bb33302c46c889b"}
Oct 30 19:57:25.328840 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:25.328402    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:25.369477 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:25.369431    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" podStartSLOduration=2.025601785 podStartE2EDuration="2.369418599s" totalImagesPullingTime="343.816814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:25.367380542 +0000 UTC m=+3193.402893006" watchObservedRunningTime="2026-10-30 19:57:25.369418599 +0000 UTC m=+3193.404931064"
Oct 30 19:57:28.644728 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:57:28.644693    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:57:31.341631 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:31.341586    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 19:57:33.522383 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:33.522341    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"]
Oct 30 19:57:33.545622 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:33.543641    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-aae4d8-8qzx4"]
Oct 30 19:57:34.097973 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:34.097942    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f0c11a-bc03-453e-b317-84b4148155ac" path="/var/lib/kubelet/pods/b9f0c11a-bc03-453e-b317-84b4148155ac/volumes"
Oct 30 19:57:46.886322 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:57:46.886286    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:57:58.644872 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:57:58.644833    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:58:01.908132 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:01.908089    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main" containerID="cri-o://9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb" gracePeriod=45
Oct 30 19:58:02.200830 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.200808    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:58:02.261913 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.261876    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\" (UniqueName: \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.262077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.261981    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.262077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.262030    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.262077 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.262065    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.262273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.262098    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.262273 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.262190    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\") pod \"82fd157c-26f2-489a-bc7c-8c48034534bb\" (UID: \"82fd157c-26f2-489a-bc7c-8c48034534bb\") "
Oct 30 19:58:02.263034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.262991    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:02.263034 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.263007    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:02.263220 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.263164    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:02.265414 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.265389    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:02.265550 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.265525    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:02.265654 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.265587    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv" pod "82fd157c-26f2-489a-bc7c-8c48034534bb" (UID: "82fd157c-26f2-489a-bc7c-8c48034534bb"). InnerVolumeSpecName "kube-api-access-wjznv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:02.363894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363864    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82fd157c-26f2-489a-bc7c-8c48034534bb-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.363894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363891    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjznv\" (UniqueName: \"kubernetes.io/projected/82fd157c-26f2-489a-bc7c-8c48034534bb-kube-api-access-wjznv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.363894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363901    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.364131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363909    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.364131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363917    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.364131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.363924    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82fd157c-26f2-489a-bc7c-8c48034534bb-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:02.775623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.775507    4320 generic.go:356] "Generic (PLEG): container finished" podID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerID="9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb" exitCode=0
Oct 30 19:58:02.775623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.775574    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" event={"ID":"82fd157c-26f2-489a-bc7c-8c48034534bb","Type":"ContainerDied","Data":"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"}
Oct 30 19:58:02.775623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.775624    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2" event={"ID":"82fd157c-26f2-489a-bc7c-8c48034534bb","Type":"ContainerDied","Data":"68c92c0624cfbb0b9e27710e4505420a36d720b4237c793df00fb22c7e30556f"}
Oct 30 19:58:02.775984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.775645    4320 scope.go:122] "RemoveContainer" containerID="9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"
Oct 30 19:58:02.775984 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.775877    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"
Oct 30 19:58:02.807017 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.806996    4320 scope.go:122] "RemoveContainer" containerID="9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"
Oct 30 19:58:02.807380 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:58:02.807353    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb\": container with ID starting with 9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb not found: ID does not exist" containerID="9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"
Oct 30 19:58:02.807474 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.807394    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb"} err="failed to get container status \"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb\": rpc error: code = NotFound desc = could not find container \"9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb\": container with ID starting with 9f631feecf2cde1cf2b0fab65a06df018b077815138ed4603ef723929cce45eb not found: ID does not exist"
Oct 30 19:58:02.844667 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.844630    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:58:02.855807 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:02.855776    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/simulator-w6-6e0c55-kserve-b87b66fc5-lb7b2"]
Oct 30 19:58:04.097657 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:04.097591    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" path="/var/lib/kubelet/pods/82fd157c-26f2-489a-bc7c-8c48034534bb/volumes"
Oct 30 19:58:09.900334 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:09.900304    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:58:28.645490 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:58:28.645454    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:58:30.738538 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.738508    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"]
Oct 30 19:58:30.739206 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.739188    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main"
Oct 30 19:58:30.739289 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.739208    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main"
Oct 30 19:58:30.739331 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.739310    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fd157c-26f2-489a-bc7c-8c48034534bb" containerName="main"
Oct 30 19:58:30.742539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.742522    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.764191 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.764166    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"]
Oct 30 19:58:30.848753 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.848721    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:30.848908 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.848849    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 19:58:30.853621 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.853578    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.853725 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.853621    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgzp\" (UniqueName: \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.853894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.853874    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.955846 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.955811    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.956015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.955946    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.956015 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.955965    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgzp\" (UniqueName: \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.956500 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.956481    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.958258 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.958231    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:30.974699 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:30.974673    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgzp\" (UniqueName: \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\") pod \"maas-api-revoke-sub-56f8ba33-fcpdm\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:31.081091 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:31.081013    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:31.388679 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:31.388649    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"]
Oct 30 19:58:31.389543 ip-10-0-130-217 kubenswrapper[4320]: W1030 19:58:31.389518    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2b0625_77ed_4731_a05e_cd80e2b1dc32.slice/crio-34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16 WatchSource:0}: Error finding container 34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16: Status 404 returned error can't find the container with id 34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16
Oct 30 19:58:32.042481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:32.042399    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm" event={"ID":"dc2b0625-77ed-4731-a05e-cd80e2b1dc32","Type":"ContainerStarted","Data":"3efa1970c3e5ab4630b228e1babbbe4f87f79997c0c61c08f6489147525b3f64"}
Oct 30 19:58:32.042481 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:32.042455    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm" event={"ID":"dc2b0625-77ed-4731-a05e-cd80e2b1dc32","Type":"ContainerStarted","Data":"34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16"}
Oct 30 19:58:32.080130 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:32.080073    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm" podStartSLOduration=2.080056258 podStartE2EDuration="2.080056258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:32.079142339 +0000 UTC m=+3260.114654803" watchObservedRunningTime="2026-10-30 19:58:32.080056258 +0000 UTC m=+3260.115568723"
Oct 30 19:58:33.052131 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:33.052096    4320 generic.go:356] "Generic (PLEG): container finished" podID="dc2b0625-77ed-4731-a05e-cd80e2b1dc32" containerID="3efa1970c3e5ab4630b228e1babbbe4f87f79997c0c61c08f6489147525b3f64" exitCode=0
Oct 30 19:58:33.052539 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:33.052138    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm" event={"ID":"dc2b0625-77ed-4731-a05e-cd80e2b1dc32","Type":"ContainerDied","Data":"3efa1970c3e5ab4630b228e1babbbe4f87f79997c0c61c08f6489147525b3f64"}
Oct 30 19:58:34.416682 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.416653    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:34.499080 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.499039    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\" (UniqueName: \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\") pod \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") "
Oct 30 19:58:34.499256 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.499196    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\") pod \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") "
Oct 30 19:58:34.499450 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.499431    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\") pod \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\" (UID: \"dc2b0625-77ed-4731-a05e-cd80e2b1dc32\") "
Oct 30 19:58:34.500930 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.500876    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca" pod "dc2b0625-77ed-4731-a05e-cd80e2b1dc32" (UID: "dc2b0625-77ed-4731-a05e-cd80e2b1dc32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:34.503199 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.503175    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp" pod "dc2b0625-77ed-4731-a05e-cd80e2b1dc32" (UID: "dc2b0625-77ed-4731-a05e-cd80e2b1dc32"). InnerVolumeSpecName "kube-api-access-zdgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.516163 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.516136    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca" pod "dc2b0625-77ed-4731-a05e-cd80e2b1dc32" (UID: "dc2b0625-77ed-4731-a05e-cd80e2b1dc32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:34.601894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.601817    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.601894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.601852    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:34.601894 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:34.601867    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgzp\" (UniqueName: \"kubernetes.io/projected/dc2b0625-77ed-4731-a05e-cd80e2b1dc32-kube-api-access-zdgzp\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:35.079247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:35.079215    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"
Oct 30 19:58:35.079247 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:35.079238    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm" event={"ID":"dc2b0625-77ed-4731-a05e-cd80e2b1dc32","Type":"ContainerDied","Data":"34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16"}
Oct 30 19:58:35.079459 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:35.079284    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b57adbb57d5f0f5694592e2a9a1f1ddccabf67065810d92b1abd9f044c5d16"
Oct 30 19:58:42.900783 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:42.900744    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:58:47.901940 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:47.901886    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" podUID="0d153b5e-947c-42ea-b996-b8582bb84528" containerName="payload-pre-processing" containerID="cri-o://efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9" gracePeriod=25
Oct 30 19:58:48.197843 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.197810    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:58:48.298318 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.298277    4320 generic.go:356] "Generic (PLEG): container finished" podID="0d153b5e-947c-42ea-b996-b8582bb84528" containerID="efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9" exitCode=0
Oct 30 19:58:48.298543 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.298520    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" event={"ID":"0d153b5e-947c-42ea-b996-b8582bb84528","Type":"ContainerDied","Data":"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"}
Oct 30 19:58:48.298759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.298578    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9" event={"ID":"0d153b5e-947c-42ea-b996-b8582bb84528","Type":"ContainerDied","Data":"6b9d9e63a41bc101dc5d97b2df0b7723eebd3f6333cf5448f3f3f059fa5661c3"}
Oct 30 19:58:48.298759 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.298647    4320 scope.go:122] "RemoveContainer" containerID="efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"
Oct 30 19:58:48.298892 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.298811    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"
Oct 30 19:58:48.313059 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.312996    4320 scope.go:122] "RemoveContainer" containerID="efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"
Oct 30 19:58:48.313371 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:58:48.313349    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9\": container with ID starting with efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9 not found: ID does not exist" containerID="efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"
Oct 30 19:58:48.313452 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.313382    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9"} err="failed to get container status \"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9\": rpc error: code = NotFound desc = could not find container \"efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9\": container with ID starting with efe1b726f70737e5e71e6b842e31be921071bb820e4573049a90de3a45c33ba9 not found: ID does not exist"
Oct 30 19:58:48.369125 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.369100    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\" (UniqueName: \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\") pod \"0d153b5e-947c-42ea-b996-b8582bb84528\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") "
Oct 30 19:58:48.369345 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.369333    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\") pod \"0d153b5e-947c-42ea-b996-b8582bb84528\" (UID: \"0d153b5e-947c-42ea-b996-b8582bb84528\") "
Oct 30 19:58:48.369730 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.369707    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume" pod "0d153b5e-947c-42ea-b996-b8582bb84528" (UID: "0d153b5e-947c-42ea-b996-b8582bb84528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:48.371101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.371075    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l" pod "0d153b5e-947c-42ea-b996-b8582bb84528" (UID: "0d153b5e-947c-42ea-b996-b8582bb84528"). InnerVolumeSpecName "kube-api-access-m9r5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:48.470745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.470690    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9r5l\" (UniqueName: \"kubernetes.io/projected/0d153b5e-947c-42ea-b996-b8582bb84528-kube-api-access-m9r5l\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:48.470745 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.470711    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d153b5e-947c-42ea-b996-b8582bb84528-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:58:48.646445 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.646420    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:58:48.653534 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.653513    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-6f4c6dd7c5-f4qc9"]
Oct 30 19:58:48.691809 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:48.691789    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:58:50.151619 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:58:50.151020    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d153b5e-947c-42ea-b996-b8582bb84528" path="/var/lib/kubelet/pods/0d153b5e-947c-42ea-b996-b8582bb84528/volumes"
Oct 30 19:58:58.644431 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:58:58.644338    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:59:28.651026 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:59:28.650982    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 19:59:33.207559 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:33.207524    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:59:33.208299 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:33.207857    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" containerID="cri-o://9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8" gracePeriod=30
Oct 30 19:59:35.506939 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:35.506898    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:59:35.508068 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:35.508023    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" containerID="cri-o://7be60f5d89534b4be827575dac500ea5cdbf8adeed12a35072d0ec25645900aa" gracePeriod=30
Oct 30 19:59:36.729755 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:36.729714    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 19:59:38.023697 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.023658    4320 patch_prober.go:29] interesting pod/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:38.024315 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.024285    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:38.174623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.174553    4320 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:38.174825 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.174635    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:38.493031 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.493008    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:59:38.547306 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547269    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547330    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547403    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547451    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547546 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547520    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547556    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547590    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547671 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547662    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.547808 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.547772    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\") pod \"e4e9348a-9635-4008-85af-6742197e476f\" (UID: \"e4e9348a-9635-4008-85af-6742197e476f\") "
Oct 30 19:59:38.549375 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.549080    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:38.549812 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.549784    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:38.549982 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.549820    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:38.550453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.550052    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:38.550453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.550228    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:38.551101 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.551067    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "kube-api-access-crj94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:38.551915 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.551891    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:38.552023 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.551913    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:38.553441 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.553421    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy" pod "e4e9348a-9635-4008-85af-6742197e476f" (UID: "e4e9348a-9635-4008-85af-6742197e476f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:38.649506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649467    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crj94\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-kube-api-access-crj94\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649504    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e9348a-9635-4008-85af-6742197e476f-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649514    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649506 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649522    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649530    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e9348a-9635-4008-85af-6742197e476f-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649538    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e9348a-9635-4008-85af-6742197e476f-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649546    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649554    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.649820 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.649561    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e9348a-9635-4008-85af-6742197e476f-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:38.742676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.742640    4320 generic.go:356] "Generic (PLEG): container finished" podID="e4e9348a-9635-4008-85af-6742197e476f" containerID="9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8" exitCode=0
Oct 30 19:59:38.742847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.742679    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" event={"ID":"e4e9348a-9635-4008-85af-6742197e476f","Type":"ContainerDied","Data":"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"}
Oct 30 19:59:38.742847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.742692    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"
Oct 30 19:59:38.742847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.742737    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z" event={"ID":"e4e9348a-9635-4008-85af-6742197e476f","Type":"ContainerDied","Data":"5afb2ee90395629585a85562e0f80c11757f3eb5060d3b4d8c3ccd29d05e031e"}
Oct 30 19:59:38.742847 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.742766    4320 scope.go:122] "RemoveContainer" containerID="9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"
Oct 30 19:59:38.753419 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.753403    4320 scope.go:122] "RemoveContainer" containerID="9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"
Oct 30 19:59:38.753701 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:59:38.753683    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8\": container with ID starting with 9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8 not found: ID does not exist" containerID="9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"
Oct 30 19:59:38.753749 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.753708    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8"} err="failed to get container status \"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8\": rpc error: code = NotFound desc = could not find container \"9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8\": container with ID starting with 9a89b68ed9202faed946c0fcc333bf1bf8e491bf879a0ed0019ce13e169dc0e8 not found: ID does not exist"
Oct 30 19:59:38.794848 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.794810    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:59:38.799226 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:38.799202    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-8vf4z"]
Oct 30 19:59:40.096281 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.096245    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e9348a-9635-4008-85af-6742197e476f" path="/var/lib/kubelet/pods/e4e9348a-9635-4008-85af-6742197e476f/volumes"
Oct 30 19:59:40.760641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.760588    4320 generic.go:356] "Generic (PLEG): container finished" podID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerID="7be60f5d89534b4be827575dac500ea5cdbf8adeed12a35072d0ec25645900aa" exitCode=0
Oct 30 19:59:40.760641 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.760624    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerDied","Data":"7be60f5d89534b4be827575dac500ea5cdbf8adeed12a35072d0ec25645900aa"}
Oct 30 19:59:40.760856 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.760686    4320 scope.go:122] "RemoveContainer" containerID="bfb769ea5d07bf565958d71d249c4417ad2b6303290646d371595dc6a0d7c537"
Oct 30 19:59:40.805567 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.805543    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:59:40.878158 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878134    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878170    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878210    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878307 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878266    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878340    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878395    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878462    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:40.878482 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878475    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878477    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:40.878676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878545    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878587    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\") pod \"72f7ad27-04e8-432d-ac33-c80fb8309095\" (UID: \"72f7ad27-04e8-432d-ac33-c80fb8309095\") "
Oct 30 19:59:40.878676 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878612    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:40.878879 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.878827    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:40.879324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.879252    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.879324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.879277    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.879324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.879291    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.879324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.879289    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:40.879324 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.879304    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.881081 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.881057    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "kube-api-access-tdcnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:40.881453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.881428    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:40.881453 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.881429    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:40.881623 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.881581    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy" pod "72f7ad27-04e8-432d-ac33-c80fb8309095" (UID: "72f7ad27-04e8-432d-ac33-c80fb8309095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:40.980981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.980949    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.980981 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.980977    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.981127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.980990    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.981127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.981002    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcnk\" (UniqueName: \"kubernetes.io/projected/72f7ad27-04e8-432d-ac33-c80fb8309095-kube-api-access-tdcnk\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:40.981127 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:40.981015    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f7ad27-04e8-432d-ac33-c80fb8309095-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 19:59:41.768873 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:41.768845    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"
Oct 30 19:59:41.769277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:41.768891    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2" event={"ID":"72f7ad27-04e8-432d-ac33-c80fb8309095","Type":"ContainerDied","Data":"984942d80187ca877943b8c000204afd6e67e272dedfc786c54efb79651364ed"}
Oct 30 19:59:41.769277 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:41.768955    4320 scope.go:122] "RemoveContainer" containerID="7be60f5d89534b4be827575dac500ea5cdbf8adeed12a35072d0ec25645900aa"
Oct 30 19:59:41.821193 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:41.821169    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:59:41.833371 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:41.833348    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-59f9ed-openshift-default-6997fbc965-f8fc2"]
Oct 30 19:59:42.098910 ip-10-0-130-217 kubenswrapper[4320]: I1030 19:59:42.098832    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" path="/var/lib/kubelet/pods/72f7ad27-04e8-432d-ac33-c80fb8309095/volumes"
Oct 30 19:59:58.645523 ip-10-0-130-217 kubenswrapper[4320]: E1030 19:59:58.645485    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:00:00.152797 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.152765    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:00.153360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153346    4320 cpu_manager.go:414] "containerMap: removing container" podUID="dc2b0625-77ed-4731-a05e-cd80e2b1dc32" containerName="revoke-keys"
Oct 30 20:00:00.153403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153362    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2b0625-77ed-4731-a05e-cd80e2b1dc32" containerName="revoke-keys"
Oct 30 20:00:00.153403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153377    4320 cpu_manager.go:414] "containerMap: removing container" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.153403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153382    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.153507 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153418    4320 cpu_manager.go:414] "containerMap: removing container" podUID="0d153b5e-947c-42ea-b996-b8582bb84528" containerName="payload-pre-processing"
Oct 30 20:00:00.153507 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153425    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d153b5e-947c-42ea-b996-b8582bb84528" containerName="payload-pre-processing"
Oct 30 20:00:00.153507 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153435    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy"
Oct 30 20:00:00.153507 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153440    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy"
Oct 30 20:00:00.153670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153528    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d153b5e-947c-42ea-b996-b8582bb84528" containerName="payload-pre-processing"
Oct 30 20:00:00.153670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153539    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e9348a-9635-4008-85af-6742197e476f" containerName="istio-proxy"
Oct 30 20:00:00.153670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153545    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.153670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153552    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2b0625-77ed-4731-a05e-cd80e2b1dc32" containerName="revoke-keys"
Oct 30 20:00:00.153670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.153557    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.157180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.157160    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:00.157318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.157298    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.157874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.157853    4320 cpu_manager.go:414] "containerMap: removing container" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.157874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.157872    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f7ad27-04e8-432d-ac33-c80fb8309095" containerName="istio-proxy"
Oct 30 20:00:00.161937 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.161920    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.175430 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.175408    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:00.179568 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.179541    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.191439 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.191417    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:00.202500 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.202395    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:00.206182 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.206160    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:00.221644 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.221582    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:00.227648 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.227581    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:00.227966 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.227863    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.231640 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.227907    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.232762 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.232733    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.233061 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.233042    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z28qt\" (UniqueName: \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.233266 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.233241    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.233396 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.233335    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmv6s\" (UniqueName: \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.248375 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.248354    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:00.251551 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.251528    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:00.272209 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.271363    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:00:00.279634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.278300    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:00.279634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.279387    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.287363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.287010    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:00.287363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.287054    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:00.287363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.287120    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.308562 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.308539    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:00.311579 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.311555    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.318553 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.314692    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:00.318553 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.314810    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.318553 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.317210    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:00.323366 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.322078    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339054    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld8xf\" (UniqueName: \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339111    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdvq\" (UniqueName: \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339156    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339190    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339268    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339318    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339346    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339476    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z28qt\" (UniqueName: \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.339634 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339544    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.340087 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.339655    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmv6s\" (UniqueName: \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.340684 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.340661    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.341008 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.340990    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.344524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.344503    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:00.344652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.344542    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:00.344715 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.344685    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.349043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.349021    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:00.349166 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.349105    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.349912 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.349873    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:00.354899 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.354753    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:00.358947 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.358679    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmv6s\" (UniqueName: \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.359615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.359577    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z28qt\" (UniqueName: \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.441290 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441222    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.441290 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441254    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.441290 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441285    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwpw\" (UniqueName: \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.441524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441326    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg699\" (UniqueName: \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.441524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441435    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.441524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441471    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.441679 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441559    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ss7\" (UniqueName: \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.441719 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441703    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.441752 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441742    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.441854 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441826    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.441919 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441896    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sp9m\" (UniqueName: \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.442002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441985    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.442038 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.441999    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.442075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442059    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld8xf\" (UniqueName: \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.442107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442097    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdvq\" (UniqueName: \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.442145 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442116    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.442183 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442154    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.442231 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442190    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.442306 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442285    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997nw\" (UniqueName: \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.442372 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442336    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2s9z\" (UniqueName: \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.442804 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.442787    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.461165 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.461139    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld8xf\" (UniqueName: \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.461563 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.461544    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\") pod \"maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.461740 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.461721    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdvq\" (UniqueName: \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.474991 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.474972    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:00.481071 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.481053    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:00.501805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.501783    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545572    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545647    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545698    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ss7\" (UniqueName: \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545769    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545803    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545848    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.545909    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sp9m\" (UniqueName: \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.546026    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.546126    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997nw\" (UniqueName: \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.546165    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2s9z\" (UniqueName: \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.546241    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwpw\" (UniqueName: \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.547334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.546282    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg699\" (UniqueName: \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.552496 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.548212    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.552496 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.548962    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.552496 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.548997    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.552496 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.549671    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.552496 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.551309    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.553264 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.553216    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.576652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.570934    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:00.576652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.572341    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:00.582373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.582344    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ss7\" (UniqueName: \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\") pod \"maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.583733 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.583704    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg699\" (UniqueName: \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\") pod \"maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.584333 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.584308    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2s9z\" (UniqueName: \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\") pod \"maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.584997 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.584967    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwpw\" (UniqueName: \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.585280 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.585225    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sp9m\" (UniqueName: \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\") pod \"maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.587715 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.587666    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997nw\" (UniqueName: \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.620404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.620212    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:00.644689 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.644478    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:00.654750 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.654536    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:00.665210 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.664235    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:00.686269 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.686036    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:00.694147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.693849    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:00.822153 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.822107    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:00.832065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.830581    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:00.845073 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:00.844442    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cdbcb2_e423_4694_8b14_9a0c6ae2e987.slice/crio-cad02ba0620b4727ba0be0bbf4d8262032bb71ce26f9f306203b305d070deb7c WatchSource:0}: Error finding container cad02ba0620b4727ba0be0bbf4d8262032bb71ce26f9f306203b305d070deb7c: Status 404 returned error can't find the container with id cad02ba0620b4727ba0be0bbf4d8262032bb71ce26f9f306203b305d070deb7c
Oct 30 20:00:00.961194 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.961071    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerStarted","Data":"7ef3094612805e4b7bf2317d8d7ee561ce40e334ea69743099f0861cc62ff5c8"}
Oct 30 20:00:00.963247 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:00.963174    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerStarted","Data":"cad02ba0620b4727ba0be0bbf4d8262032bb71ce26f9f306203b305d070deb7c"}
Oct 30 20:00:01.043910 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.043874    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:01.075238 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.075208    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b259aa_f4d5_46c0_be7e_3b95c53e13ae.slice/crio-8431c5e40bf87d59daec8faf9d3bf564ca9cb3f24ffd3d0986594143d7679592 WatchSource:0}: Error finding container 8431c5e40bf87d59daec8faf9d3bf564ca9cb3f24ffd3d0986594143d7679592: Status 404 returned error can't find the container with id 8431c5e40bf87d59daec8faf9d3bf564ca9cb3f24ffd3d0986594143d7679592
Oct 30 20:00:01.140702 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.140656    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1eb04e2_4ee0_4fc0_bdc8_9db4110552ba.slice/crio-a7a9653307aa6bf3a1fb52f2ac57f437eb71b6d8ee16ff50eb37c747d7f3ae0d WatchSource:0}: Error finding container a7a9653307aa6bf3a1fb52f2ac57f437eb71b6d8ee16ff50eb37c747d7f3ae0d: Status 404 returned error can't find the container with id a7a9653307aa6bf3a1fb52f2ac57f437eb71b6d8ee16ff50eb37c747d7f3ae0d
Oct 30 20:00:01.142859 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.142834    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:01.159643 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.159618    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:01.160193 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.160174    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec81397_7f07_43b8_a73f_acae90decc27.slice/crio-9aac4b28e9eb967add8c914ec7302431f4dac6caa09795ac9959f9e3577c5adf WatchSource:0}: Error finding container 9aac4b28e9eb967add8c914ec7302431f4dac6caa09795ac9959f9e3577c5adf: Status 404 returned error can't find the container with id 9aac4b28e9eb967add8c914ec7302431f4dac6caa09795ac9959f9e3577c5adf
Oct 30 20:00:01.180206 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.180110    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:01.182574 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.182550    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9cfbc6_f03b_4a46_adf5_9da7ab25a150.slice/crio-a703dc5723ff9b03e36fa38b505028c5e41add85680c35df3dba26c6e45692f0 WatchSource:0}: Error finding container a703dc5723ff9b03e36fa38b505028c5e41add85680c35df3dba26c6e45692f0: Status 404 returned error can't find the container with id a703dc5723ff9b03e36fa38b505028c5e41add85680c35df3dba26c6e45692f0
Oct 30 20:00:01.273356 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.273316    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ebcd68_f7db_4238_87db_38b0c1b83d3a.slice/crio-f0063fc801911c0ac065abea76a0058d4b3f0f15610113a063624d3ee35abf3d WatchSource:0}: Error finding container f0063fc801911c0ac065abea76a0058d4b3f0f15610113a063624d3ee35abf3d: Status 404 returned error can't find the container with id f0063fc801911c0ac065abea76a0058d4b3f0f15610113a063624d3ee35abf3d
Oct 30 20:00:01.274881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.274840    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:01.308760 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.308562    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:01.314741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.314720    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:01.320295 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.320274    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:01.324922 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.324900    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:01.374968 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.374938    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2371dc5c_ddc4_44f9_a83f_d2753aa58873.slice/crio-5ac347f4fb939c6ce5057221e1b44e281e6d4f3a1a8a2afe90165cc91b151732 WatchSource:0}: Error finding container 5ac347f4fb939c6ce5057221e1b44e281e6d4f3a1a8a2afe90165cc91b151732: Status 404 returned error can't find the container with id 5ac347f4fb939c6ce5057221e1b44e281e6d4f3a1a8a2afe90165cc91b151732
Oct 30 20:00:01.376273 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.376247    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef1776d_7576_4554_94e0_30b201c7009c.slice/crio-72fe363744fdc7cdc26a33d9dfd820e6441b02f6df27179d7d050c9cb8cb9082 WatchSource:0}: Error finding container 72fe363744fdc7cdc26a33d9dfd820e6441b02f6df27179d7d050c9cb8cb9082: Status 404 returned error can't find the container with id 72fe363744fdc7cdc26a33d9dfd820e6441b02f6df27179d7d050c9cb8cb9082
Oct 30 20:00:01.377805 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.377781    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55e4284_8465_45aa_b169_056a83859ed4.slice/crio-e762c00dba5fa319bf876ed56bac20f9cbf96f0ef1e15f8ec7ff2e4f8087b92b WatchSource:0}: Error finding container e762c00dba5fa319bf876ed56bac20f9cbf96f0ef1e15f8ec7ff2e4f8087b92b: Status 404 returned error can't find the container with id e762c00dba5fa319bf876ed56bac20f9cbf96f0ef1e15f8ec7ff2e4f8087b92b
Oct 30 20:00:01.471418 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:00:01.471352    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf881ead9_5adb_443c_aabb_8792af9e5cca.slice/crio-2094bc3a48f37db0501d14c5998596e7f0ba34be4e857be6d95aaf2243c29709 WatchSource:0}: Error finding container 2094bc3a48f37db0501d14c5998596e7f0ba34be4e857be6d95aaf2243c29709: Status 404 returned error can't find the container with id 2094bc3a48f37db0501d14c5998596e7f0ba34be4e857be6d95aaf2243c29709
Oct 30 20:00:01.973832 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.973798    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerStarted","Data":"5ac347f4fb939c6ce5057221e1b44e281e6d4f3a1a8a2afe90165cc91b151732"}
Oct 30 20:00:01.975379 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.975348    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerStarted","Data":"e762c00dba5fa319bf876ed56bac20f9cbf96f0ef1e15f8ec7ff2e4f8087b92b"}
Oct 30 20:00:01.977562 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.977539    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerStarted","Data":"7ab9fb8e32f1790c950b4414c05cca02d779c5080d77b22bef29bc34f4c090c5"}
Oct 30 20:00:01.979359 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.979334    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerStarted","Data":"d4e2bf0c6976ba66639147b14c12d2ad2dfaa369f47481cb737ca670223e94c3"}
Oct 30 20:00:01.979486 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.979380    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerStarted","Data":"8431c5e40bf87d59daec8faf9d3bf564ca9cb3f24ffd3d0986594143d7679592"}
Oct 30 20:00:01.981146 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.981125    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerStarted","Data":"a721a1883365336238c39b25d3e6d009d94b06f2cf23828db1a154557e4c0d23"}
Oct 30 20:00:01.981237 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.981160    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerStarted","Data":"a7a9653307aa6bf3a1fb52f2ac57f437eb71b6d8ee16ff50eb37c747d7f3ae0d"}
Oct 30 20:00:01.982417 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.982397    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerStarted","Data":"2094bc3a48f37db0501d14c5998596e7f0ba34be4e857be6d95aaf2243c29709"}
Oct 30 20:00:01.983640 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.983615    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerStarted","Data":"72fe363744fdc7cdc26a33d9dfd820e6441b02f6df27179d7d050c9cb8cb9082"}
Oct 30 20:00:01.985312 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.985286    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerStarted","Data":"407f892d83f273c0df68b945cfc11de6782e7fa2d4ab6fcbdb696c97285774ed"}
Oct 30 20:00:01.985407 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.985323    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerStarted","Data":"9aac4b28e9eb967add8c914ec7302431f4dac6caa09795ac9959f9e3577c5adf"}
Oct 30 20:00:01.986951 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.986928    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerStarted","Data":"c514ea896ba57c5e32cb6e64045f1b9cae6cf8b8b3fe6af11d2ff00ab36bb945"}
Oct 30 20:00:01.987062 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.986964    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerStarted","Data":"a703dc5723ff9b03e36fa38b505028c5e41add85680c35df3dba26c6e45692f0"}
Oct 30 20:00:01.988488 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.988470    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerStarted","Data":"834caeda729ee5e28ae02289cef5b4424669beb8f4558d3425ff73c10be39ba4"}
Oct 30 20:00:01.988788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.988769    4320 scope.go:122] "RemoveContainer" containerID="834caeda729ee5e28ae02289cef5b4424669beb8f4558d3425ff73c10be39ba4"
Oct 30 20:00:01.989770 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:01.989745    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerStarted","Data":"f0063fc801911c0ac065abea76a0058d4b3f0f15610113a063624d3ee35abf3d"}
Oct 30 20:00:02.015371 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.015330    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" podStartSLOduration=2.015317973 podStartE2EDuration="2.015317973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.013344871 +0000 UTC m=+3350.048857334" watchObservedRunningTime="2026-10-30 20:00:02.015317973 +0000 UTC m=+3350.050830437"
Oct 30 20:00:02.093769 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.093719    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" podStartSLOduration=2.093703732 podStartE2EDuration="2.093703732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.09069347 +0000 UTC m=+3350.126205935" watchObservedRunningTime="2026-10-30 20:00:02.093703732 +0000 UTC m=+3350.129216197"
Oct 30 20:00:02.094270 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.094232    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" podStartSLOduration=2.094223122 podStartE2EDuration="2.094223122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.048987257 +0000 UTC m=+3350.084499721" watchObservedRunningTime="2026-10-30 20:00:02.094223122 +0000 UTC m=+3350.129735642"
Oct 30 20:00:02.170518 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.170465    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" podStartSLOduration=2.170451704 podStartE2EDuration="2.170451704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.168386404 +0000 UTC m=+3350.203898868" watchObservedRunningTime="2026-10-30 20:00:02.170451704 +0000 UTC m=+3350.205964201"
Oct 30 20:00:02.210403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.210350    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" podStartSLOduration=2.210335247 podStartE2EDuration="2.210335247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.208100366 +0000 UTC m=+3350.243612833" watchObservedRunningTime="2026-10-30 20:00:02.210335247 +0000 UTC m=+3350.245847710"
Oct 30 20:00:02.999288 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.999259    4320 generic.go:356] "Generic (PLEG): container finished" podID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerID="7ab9fb8e32f1790c950b4414c05cca02d779c5080d77b22bef29bc34f4c090c5" exitCode=60
Oct 30 20:00:02.999399 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.999343    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerDied","Data":"7ab9fb8e32f1790c950b4414c05cca02d779c5080d77b22bef29bc34f4c090c5"}
Oct 30 20:00:02.999554 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:02.999539    4320 scope.go:122] "RemoveContainer" containerID="7ab9fb8e32f1790c950b4414c05cca02d779c5080d77b22bef29bc34f4c090c5"
Oct 30 20:00:03.001253 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.001229    4320 generic.go:356] "Generic (PLEG): container finished" podID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerID="d4e2bf0c6976ba66639147b14c12d2ad2dfaa369f47481cb737ca670223e94c3" exitCode=60
Oct 30 20:00:03.001331 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.001307    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerDied","Data":"d4e2bf0c6976ba66639147b14c12d2ad2dfaa369f47481cb737ca670223e94c3"}
Oct 30 20:00:03.001523 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.001506    4320 scope.go:122] "RemoveContainer" containerID="d4e2bf0c6976ba66639147b14c12d2ad2dfaa369f47481cb737ca670223e94c3"
Oct 30 20:00:03.003115 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.003093    4320 generic.go:356] "Generic (PLEG): container finished" podID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerID="a721a1883365336238c39b25d3e6d009d94b06f2cf23828db1a154557e4c0d23" exitCode=60
Oct 30 20:00:03.003243 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.003215    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerDied","Data":"a721a1883365336238c39b25d3e6d009d94b06f2cf23828db1a154557e4c0d23"}
Oct 30 20:00:03.003351 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.003333    4320 scope.go:122] "RemoveContainer" containerID="a721a1883365336238c39b25d3e6d009d94b06f2cf23828db1a154557e4c0d23"
Oct 30 20:00:03.004996 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.004978    4320 generic.go:356] "Generic (PLEG): container finished" podID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerID="0cdefa1b7fa112a9d3ea8bc020d132e8fbf5a6a4cc1b2c0181d562f94687e861" exitCode=60
Oct 30 20:00:03.005070 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.005053    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerDied","Data":"0cdefa1b7fa112a9d3ea8bc020d132e8fbf5a6a4cc1b2c0181d562f94687e861"}
Oct 30 20:00:03.005267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.005251    4320 scope.go:122] "RemoveContainer" containerID="0cdefa1b7fa112a9d3ea8bc020d132e8fbf5a6a4cc1b2c0181d562f94687e861"
Oct 30 20:00:03.006665 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.006642    4320 generic.go:356] "Generic (PLEG): container finished" podID="6ef1776d-7576-4554-94e0-30b201c7009c" containerID="f7391e5d171b15632f389a3fce19263dadf53c39a851479fef637e7572c99c5f" exitCode=60
Oct 30 20:00:03.006761 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.006723    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerDied","Data":"f7391e5d171b15632f389a3fce19263dadf53c39a851479fef637e7572c99c5f"}
Oct 30 20:00:03.006900 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.006883    4320 scope.go:122] "RemoveContainer" containerID="f7391e5d171b15632f389a3fce19263dadf53c39a851479fef637e7572c99c5f"
Oct 30 20:00:03.008357 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.008339    4320 generic.go:356] "Generic (PLEG): container finished" podID="bec81397-7f07-43b8-a73f-acae90decc27" containerID="407f892d83f273c0df68b945cfc11de6782e7fa2d4ab6fcbdb696c97285774ed" exitCode=60
Oct 30 20:00:03.008455 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.008372    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerDied","Data":"407f892d83f273c0df68b945cfc11de6782e7fa2d4ab6fcbdb696c97285774ed"}
Oct 30 20:00:03.008689 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.008671    4320 scope.go:122] "RemoveContainer" containerID="407f892d83f273c0df68b945cfc11de6782e7fa2d4ab6fcbdb696c97285774ed"
Oct 30 20:00:03.012069 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.012033    4320 generic.go:356] "Generic (PLEG): container finished" podID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerID="c514ea896ba57c5e32cb6e64045f1b9cae6cf8b8b3fe6af11d2ff00ab36bb945" exitCode=60
Oct 30 20:00:03.012280 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.012256    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerDied","Data":"c514ea896ba57c5e32cb6e64045f1b9cae6cf8b8b3fe6af11d2ff00ab36bb945"}
Oct 30 20:00:03.012390 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.012374    4320 scope.go:122] "RemoveContainer" containerID="c514ea896ba57c5e32cb6e64045f1b9cae6cf8b8b3fe6af11d2ff00ab36bb945"
Oct 30 20:00:03.014477 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.014455    4320 generic.go:356] "Generic (PLEG): container finished" podID="82404b8d-352d-487b-991f-5931d7c74b1a" containerID="834caeda729ee5e28ae02289cef5b4424669beb8f4558d3425ff73c10be39ba4" exitCode=60
Oct 30 20:00:03.014552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.014520    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerDied","Data":"834caeda729ee5e28ae02289cef5b4424669beb8f4558d3425ff73c10be39ba4"}
Oct 30 20:00:03.016404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.016378    4320 generic.go:356] "Generic (PLEG): container finished" podID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerID="a0889414ce385e81a3d2cd33d4d0c051f8fda0e8beda42cfd42a34d969df2f9e" exitCode=60
Oct 30 20:00:03.016528 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.016508    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerDied","Data":"a0889414ce385e81a3d2cd33d4d0c051f8fda0e8beda42cfd42a34d969df2f9e"}
Oct 30 20:00:03.016674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.016656    4320 scope.go:122] "RemoveContainer" containerID="a0889414ce385e81a3d2cd33d4d0c051f8fda0e8beda42cfd42a34d969df2f9e"
Oct 30 20:00:03.019686 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.019668    4320 generic.go:356] "Generic (PLEG): container finished" podID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerID="39c9b2585746ad612385a57ff3f405c209882b75a65ad3f4f2a9fadb2a9bbb42" exitCode=60
Oct 30 20:00:03.019786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.019747    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerDied","Data":"39c9b2585746ad612385a57ff3f405c209882b75a65ad3f4f2a9fadb2a9bbb42"}
Oct 30 20:00:03.020065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.020045    4320 scope.go:122] "RemoveContainer" containerID="39c9b2585746ad612385a57ff3f405c209882b75a65ad3f4f2a9fadb2a9bbb42"
Oct 30 20:00:03.022233 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.022218    4320 generic.go:356] "Generic (PLEG): container finished" podID="a55e4284-8465-45aa-b169-056a83859ed4" containerID="9b5dcf341dd8c61a51e68add3364152de9dea438249894dadce05536c6868cb4" exitCode=60
Oct 30 20:00:03.022287 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.022246    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerDied","Data":"9b5dcf341dd8c61a51e68add3364152de9dea438249894dadce05536c6868cb4"}
Oct 30 20:00:03.022505 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:03.022488    4320 scope.go:122] "RemoveContainer" containerID="9b5dcf341dd8c61a51e68add3364152de9dea438249894dadce05536c6868cb4"
Oct 30 20:00:04.032936 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.032891    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerStarted","Data":"a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"}
Oct 30 20:00:04.035102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.035076    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerStarted","Data":"a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"}
Oct 30 20:00:04.037397 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.037365    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerStarted","Data":"58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"}
Oct 30 20:00:04.039369 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.039347    4320 generic.go:356] "Generic (PLEG): container finished" podID="82404b8d-352d-487b-991f-5931d7c74b1a" containerID="b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4" exitCode=60
Oct 30 20:00:04.039465 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.039420    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerDied","Data":"b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4"}
Oct 30 20:00:04.039528 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.039469    4320 scope.go:122] "RemoveContainer" containerID="834caeda729ee5e28ae02289cef5b4424669beb8f4558d3425ff73c10be39ba4"
Oct 30 20:00:04.039768 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.039749    4320 scope.go:122] "RemoveContainer" containerID="b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4"
Oct 30 20:00:04.040078 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:04.040040    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr_odh-ai-gateway-infra(82404b8d-352d-487b-991f-5931d7c74b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" podUID="82404b8d-352d-487b-991f-5931d7c74b1a"
Oct 30 20:00:04.041477 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.041457    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerStarted","Data":"8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"}
Oct 30 20:00:04.043343 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.043306    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerStarted","Data":"0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"}
Oct 30 20:00:04.045348 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.045327    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerStarted","Data":"fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"}
Oct 30 20:00:04.047418 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.047396    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerStarted","Data":"f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"}
Oct 30 20:00:04.049371 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.049351    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerStarted","Data":"1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"}
Oct 30 20:00:04.051281 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.051259    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerStarted","Data":"e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"}
Oct 30 20:00:04.053271 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.053250    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerStarted","Data":"27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"}
Oct 30 20:00:04.094335 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.090250    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" podStartSLOduration=4.090230943 podStartE2EDuration="4.090230943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.087214726 +0000 UTC m=+3352.122727191" watchObservedRunningTime="2026-10-30 20:00:04.090230943 +0000 UTC m=+3352.125743408"
Oct 30 20:00:04.197098 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.197047    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" podStartSLOduration=4.19703253 podStartE2EDuration="4.19703253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.190927224 +0000 UTC m=+3352.226439685" watchObservedRunningTime="2026-10-30 20:00:04.19703253 +0000 UTC m=+3352.232545005"
Oct 30 20:00:04.275227 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.275168    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" podStartSLOduration=4.275150689 podStartE2EDuration="4.275150689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.272846221 +0000 UTC m=+3352.308358685" watchObservedRunningTime="2026-10-30 20:00:04.275150689 +0000 UTC m=+3352.310663154"
Oct 30 20:00:04.427029 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.426983    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" podStartSLOduration=4.42696888 podStartE2EDuration="4.42696888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.424456038 +0000 UTC m=+3352.459968503" watchObservedRunningTime="2026-10-30 20:00:04.42696888 +0000 UTC m=+3352.462481343"
Oct 30 20:00:04.515753 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:04.515708    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" podStartSLOduration=4.515693991 podStartE2EDuration="4.515693991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.512737162 +0000 UTC m=+3352.548249625" watchObservedRunningTime="2026-10-30 20:00:04.515693991 +0000 UTC m=+3352.551206455"
Oct 30 20:00:05.061658 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.061568    4320 generic.go:356] "Generic (PLEG): container finished" podID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerID="f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f" exitCode=60
Oct 30 20:00:05.062107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.061647    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerDied","Data":"f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"}
Oct 30 20:00:05.062107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.061711    4320 scope.go:122] "RemoveContainer" containerID="7ab9fb8e32f1790c950b4414c05cca02d779c5080d77b22bef29bc34f4c090c5"
Oct 30 20:00:05.062107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.061956    4320 scope.go:122] "RemoveContainer" containerID="f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"
Oct 30 20:00:05.062264 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.062237    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5_odh-ai-gateway-infra(95cdbcb2-e423-4694-8b14-9a0c6ae2e987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987"
Oct 30 20:00:05.063805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.063783    4320 generic.go:356] "Generic (PLEG): container finished" podID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerID="1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e" exitCode=60
Oct 30 20:00:05.063898 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.063845    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerDied","Data":"1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"}
Oct 30 20:00:05.064074 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.064058    4320 scope.go:122] "RemoveContainer" containerID="1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"
Oct 30 20:00:05.064289 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.064271    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz_odh-ai-gateway-infra(c2b259aa-f4d5-46c0-be7e-3b95c53e13ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae"
Oct 30 20:00:05.065660 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.065629    4320 generic.go:356] "Generic (PLEG): container finished" podID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerID="e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75" exitCode=60
Oct 30 20:00:05.065733 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.065709    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerDied","Data":"e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"}
Oct 30 20:00:05.065982 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.065953    4320 scope.go:122] "RemoveContainer" containerID="e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"
Oct 30 20:00:05.066217 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.066198    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6_odh-ai-gateway-infra(f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba"
Oct 30 20:00:05.067671 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.067646    4320 generic.go:356] "Generic (PLEG): container finished" podID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerID="27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb" exitCode=60
Oct 30 20:00:05.067774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.067724    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerDied","Data":"27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"}
Oct 30 20:00:05.067919 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.067903    4320 scope.go:122] "RemoveContainer" containerID="27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"
Oct 30 20:00:05.068150 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.068132    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx_odh-ai-gateway-infra(f881ead9-5adb-443c-aabb-8792af9e5cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca"
Oct 30 20:00:05.069661 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.069642    4320 generic.go:356] "Generic (PLEG): container finished" podID="6ef1776d-7576-4554-94e0-30b201c7009c" containerID="a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013" exitCode=60
Oct 30 20:00:05.069770 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.069712    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerDied","Data":"a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"}
Oct 30 20:00:05.069937 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.069911    4320 scope.go:122] "RemoveContainer" containerID="a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"
Oct 30 20:00:05.070172 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.070148    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq_odh-ai-gateway-infra(6ef1776d-7576-4554-94e0-30b201c7009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" podUID="6ef1776d-7576-4554-94e0-30b201c7009c"
Oct 30 20:00:05.071711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.071688    4320 generic.go:356] "Generic (PLEG): container finished" podID="bec81397-7f07-43b8-a73f-acae90decc27" containerID="a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694" exitCode=60
Oct 30 20:00:05.071828 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.071787    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerDied","Data":"a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"}
Oct 30 20:00:05.071970 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.071954    4320 scope.go:122] "RemoveContainer" containerID="a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"
Oct 30 20:00:05.072175 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.072152    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw_odh-ai-gateway-infra(bec81397-7f07-43b8-a73f-acae90decc27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" podUID="bec81397-7f07-43b8-a73f-acae90decc27"
Oct 30 20:00:05.073873 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.073853    4320 generic.go:356] "Generic (PLEG): container finished" podID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerID="58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f" exitCode=60
Oct 30 20:00:05.073979 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.073931    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerDied","Data":"58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"}
Oct 30 20:00:05.074128 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.074111    4320 scope.go:122] "RemoveContainer" containerID="58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"
Oct 30 20:00:05.074328 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.074307    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd_odh-ai-gateway-infra(5d9cfbc6-f03b-4a46-adf5-9da7ab25a150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150"
Oct 30 20:00:05.075918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.075891    4320 scope.go:122] "RemoveContainer" containerID="b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4"
Oct 30 20:00:05.076150 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.076130    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr_odh-ai-gateway-infra(82404b8d-352d-487b-991f-5931d7c74b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" podUID="82404b8d-352d-487b-991f-5931d7c74b1a"
Oct 30 20:00:05.076216 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.076180    4320 scope.go:122] "RemoveContainer" containerID="d4e2bf0c6976ba66639147b14c12d2ad2dfaa369f47481cb737ca670223e94c3"
Oct 30 20:00:05.077358 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.077341    4320 generic.go:356] "Generic (PLEG): container finished" podID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerID="8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089" exitCode=60
Oct 30 20:00:05.077475 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.077453    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerDied","Data":"8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"}
Oct 30 20:00:05.077607 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.077583    4320 scope.go:122] "RemoveContainer" containerID="8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"
Oct 30 20:00:05.077916 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.077888    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh_odh-ai-gateway-infra(37ebcd68-f7db-4238-87db-38b0c1b83d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a"
Oct 30 20:00:05.079320 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.079296    4320 generic.go:356] "Generic (PLEG): container finished" podID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerID="0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2" exitCode=60
Oct 30 20:00:05.079402 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.079370    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerDied","Data":"0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"}
Oct 30 20:00:05.079551 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.079536    4320 scope.go:122] "RemoveContainer" containerID="0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"
Oct 30 20:00:05.079802 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.079767    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f_odh-ai-gateway-infra(2371dc5c-ddc4-44f9-a83f-d2753aa58873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873"
Oct 30 20:00:05.081392 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.081368    4320 generic.go:356] "Generic (PLEG): container finished" podID="a55e4284-8465-45aa-b169-056a83859ed4" containerID="fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3" exitCode=60
Oct 30 20:00:05.081528 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.081454    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerDied","Data":"fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"}
Oct 30 20:00:05.081937 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.081921    4320 scope.go:122] "RemoveContainer" containerID="fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"
Oct 30 20:00:05.082197 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:05.082175    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z_odh-ai-gateway-infra(a55e4284-8465-45aa-b169-056a83859ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" podUID="a55e4284-8465-45aa-b169-056a83859ed4"
Oct 30 20:00:05.094729 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.094706    4320 scope.go:122] "RemoveContainer" containerID="a721a1883365336238c39b25d3e6d009d94b06f2cf23828db1a154557e4c0d23"
Oct 30 20:00:05.108215 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.108195    4320 scope.go:122] "RemoveContainer" containerID="0cdefa1b7fa112a9d3ea8bc020d132e8fbf5a6a4cc1b2c0181d562f94687e861"
Oct 30 20:00:05.121115 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.121100    4320 scope.go:122] "RemoveContainer" containerID="f7391e5d171b15632f389a3fce19263dadf53c39a851479fef637e7572c99c5f"
Oct 30 20:00:05.133532 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.133505    4320 scope.go:122] "RemoveContainer" containerID="407f892d83f273c0df68b945cfc11de6782e7fa2d4ab6fcbdb696c97285774ed"
Oct 30 20:00:05.152445 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.152427    4320 scope.go:122] "RemoveContainer" containerID="c514ea896ba57c5e32cb6e64045f1b9cae6cf8b8b3fe6af11d2ff00ab36bb945"
Oct 30 20:00:05.164551 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.164535    4320 scope.go:122] "RemoveContainer" containerID="a0889414ce385e81a3d2cd33d4d0c051f8fda0e8beda42cfd42a34d969df2f9e"
Oct 30 20:00:05.177179 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.177137    4320 scope.go:122] "RemoveContainer" containerID="39c9b2585746ad612385a57ff3f405c209882b75a65ad3f4f2a9fadb2a9bbb42"
Oct 30 20:00:05.190436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:05.190419    4320 scope.go:122] "RemoveContainer" containerID="9b5dcf341dd8c61a51e68add3364152de9dea438249894dadce05536c6868cb4"
Oct 30 20:00:06.112957 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.112926    4320 scope.go:122] "RemoveContainer" containerID="f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"
Oct 30 20:00:06.113382 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.113201    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5_odh-ai-gateway-infra(95cdbcb2-e423-4694-8b14-9a0c6ae2e987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987"
Oct 30 20:00:06.114943 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.114920    4320 scope.go:122] "RemoveContainer" containerID="1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"
Oct 30 20:00:06.115198 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.115174    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz_odh-ai-gateway-infra(c2b259aa-f4d5-46c0-be7e-3b95c53e13ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae"
Oct 30 20:00:06.116516 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.116498    4320 scope.go:122] "RemoveContainer" containerID="e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"
Oct 30 20:00:06.116784 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.116762    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6_odh-ai-gateway-infra(f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba"
Oct 30 20:00:06.118050 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.118032    4320 scope.go:122] "RemoveContainer" containerID="27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"
Oct 30 20:00:06.118306 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.118279    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx_odh-ai-gateway-infra(f881ead9-5adb-443c-aabb-8792af9e5cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca"
Oct 30 20:00:06.119653 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.119634    4320 scope.go:122] "RemoveContainer" containerID="a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"
Oct 30 20:00:06.119885 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.119863    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq_odh-ai-gateway-infra(6ef1776d-7576-4554-94e0-30b201c7009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" podUID="6ef1776d-7576-4554-94e0-30b201c7009c"
Oct 30 20:00:06.121154 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.121138    4320 scope.go:122] "RemoveContainer" containerID="a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"
Oct 30 20:00:06.121381 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.121357    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw_odh-ai-gateway-infra(bec81397-7f07-43b8-a73f-acae90decc27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" podUID="bec81397-7f07-43b8-a73f-acae90decc27"
Oct 30 20:00:06.122740 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.122718    4320 scope.go:122] "RemoveContainer" containerID="58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"
Oct 30 20:00:06.122948 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.122926    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd_odh-ai-gateway-infra(5d9cfbc6-f03b-4a46-adf5-9da7ab25a150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150"
Oct 30 20:00:06.124284 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.124265    4320 scope.go:122] "RemoveContainer" containerID="8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"
Oct 30 20:00:06.124494 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.124469    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh_odh-ai-gateway-infra(37ebcd68-f7db-4238-87db-38b0c1b83d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a"
Oct 30 20:00:06.125937 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.125918    4320 scope.go:122] "RemoveContainer" containerID="0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"
Oct 30 20:00:06.126140 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.126117    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f_odh-ai-gateway-infra(2371dc5c-ddc4-44f9-a83f-d2753aa58873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873"
Oct 30 20:00:06.127542 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:06.127524    4320 scope.go:122] "RemoveContainer" containerID="fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"
Oct 30 20:00:06.127764 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:06.127746    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z_odh-ai-gateway-infra(a55e4284-8465-45aa-b169-056a83859ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" podUID="a55e4284-8465-45aa-b169-056a83859ed4"
Oct 30 20:00:13.376696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:13.376655    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:00:15.089073 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:15.089024    4320 scope.go:122] "RemoveContainer" containerID="a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"
Oct 30 20:00:15.089555 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:15.089119    4320 scope.go:122] "RemoveContainer" containerID="b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4"
Oct 30 20:00:15.089555 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:15.089180    4320 scope.go:122] "RemoveContainer" containerID="58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"
Oct 30 20:00:15.089555 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:15.089265    4320 scope.go:122] "RemoveContainer" containerID="0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"
Oct 30 20:00:15.089555 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:15.089488    4320 scope.go:122] "RemoveContainer" containerID="e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"
Oct 30 20:00:16.089853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.089822    4320 scope.go:122] "RemoveContainer" containerID="fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"
Oct 30 20:00:16.273677 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.273641    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerStarted","Data":"43b6a3341af30b48d959d73b444631b350c528ff77260c4898a42773a375084f"}
Oct 30 20:00:16.276897 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.276872    4320 generic.go:356] "Generic (PLEG): container finished" podID="bec81397-7f07-43b8-a73f-acae90decc27" containerID="850060dba9783f5d173494988fdf2d0689c3d1f4e6af7212fbb0a904407045bb" exitCode=60
Oct 30 20:00:16.277009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.276945    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerDied","Data":"850060dba9783f5d173494988fdf2d0689c3d1f4e6af7212fbb0a904407045bb"}
Oct 30 20:00:16.277009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.277007    4320 scope.go:122] "RemoveContainer" containerID="a0c8293f17c51de581709995b437e848f8742404a90d891004f6945af241f694"
Oct 30 20:00:16.277409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.277387    4320 scope.go:122] "RemoveContainer" containerID="850060dba9783f5d173494988fdf2d0689c3d1f4e6af7212fbb0a904407045bb"
Oct 30 20:00:16.277862 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:16.277834    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw_odh-ai-gateway-infra(bec81397-7f07-43b8-a73f-acae90decc27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" podUID="bec81397-7f07-43b8-a73f-acae90decc27"
Oct 30 20:00:16.279576 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.279551    4320 generic.go:356] "Generic (PLEG): container finished" podID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerID="1344fdf269b54609fc9c81d0e24a0b7b8409a33a68e95366439a2b691a4f10c7" exitCode=60
Oct 30 20:00:16.279710 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.279623    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerDied","Data":"1344fdf269b54609fc9c81d0e24a0b7b8409a33a68e95366439a2b691a4f10c7"}
Oct 30 20:00:16.279969 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.279942    4320 scope.go:122] "RemoveContainer" containerID="1344fdf269b54609fc9c81d0e24a0b7b8409a33a68e95366439a2b691a4f10c7"
Oct 30 20:00:16.280271 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:16.280249    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd_odh-ai-gateway-infra(5d9cfbc6-f03b-4a46-adf5-9da7ab25a150)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150"
Oct 30 20:00:16.282135 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.282111    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerStarted","Data":"6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5"}
Oct 30 20:00:16.282481 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.282463    4320 scope.go:122] "RemoveContainer" containerID="6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5"
Oct 30 20:00:16.282764 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:16.282739    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr_odh-ai-gateway-infra(82404b8d-352d-487b-991f-5931d7c74b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" podUID="82404b8d-352d-487b-991f-5931d7c74b1a"
Oct 30 20:00:16.284478 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.284459    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerStarted","Data":"76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780"}
Oct 30 20:00:16.284835 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.284809    4320 scope.go:122] "RemoveContainer" containerID="76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780"
Oct 30 20:00:16.285093 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:16.285071    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f_odh-ai-gateway-infra(2371dc5c-ddc4-44f9-a83f-d2753aa58873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873"
Oct 30 20:00:16.480910 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:16.480888    4320 scope.go:122] "RemoveContainer" containerID="58e8cb96d176f083c9aa40e09a1d48b016ff0a5e79ee26d05e73954a85c2d28f"
Oct 30 20:00:17.089074 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.089053    4320 scope.go:122] "RemoveContainer" containerID="f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"
Oct 30 20:00:17.089219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.089201    4320 scope.go:122] "RemoveContainer" containerID="a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"
Oct 30 20:00:17.326064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.325990    4320 generic.go:356] "Generic (PLEG): container finished" podID="82404b8d-352d-487b-991f-5931d7c74b1a" containerID="6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5" exitCode=60
Oct 30 20:00:17.326509 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.326063    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerDied","Data":"6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5"}
Oct 30 20:00:17.326509 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.326130    4320 scope.go:122] "RemoveContainer" containerID="b6993a6f05a98f17ecbdce56c963e62249bbc192bfdc87af92166ec45202d3b4"
Oct 30 20:00:17.326672 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.326612    4320 scope.go:122] "RemoveContainer" containerID="6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5"
Oct 30 20:00:17.327889 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:17.327864    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr_odh-ai-gateway-infra(82404b8d-352d-487b-991f-5931d7c74b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" podUID="82404b8d-352d-487b-991f-5931d7c74b1a"
Oct 30 20:00:17.328220 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.328196    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:17.329306 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.329279    4320 generic.go:356] "Generic (PLEG): container finished" podID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerID="76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780" exitCode=60
Oct 30 20:00:17.329422 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.329344    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerDied","Data":"76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780"}
Oct 30 20:00:17.329762 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.329744    4320 scope.go:122] "RemoveContainer" containerID="76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780"
Oct 30 20:00:17.330111 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:17.330086    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f_odh-ai-gateway-infra(2371dc5c-ddc4-44f9-a83f-d2753aa58873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873"
Oct 30 20:00:17.332541 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.332518    4320 generic.go:356] "Generic (PLEG): container finished" podID="a55e4284-8465-45aa-b169-056a83859ed4" containerID="6bf02228513481f59a7ece7f687335e412fefb432fdf08f972b325e46afeecb5" exitCode=60
Oct 30 20:00:17.332674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.332614    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerDied","Data":"6bf02228513481f59a7ece7f687335e412fefb432fdf08f972b325e46afeecb5"}
Oct 30 20:00:17.333069 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.333048    4320 scope.go:122] "RemoveContainer" containerID="6bf02228513481f59a7ece7f687335e412fefb432fdf08f972b325e46afeecb5"
Oct 30 20:00:17.333446 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:17.333422    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z_odh-ai-gateway-infra(a55e4284-8465-45aa-b169-056a83859ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" podUID="a55e4284-8465-45aa-b169-056a83859ed4"
Oct 30 20:00:17.334588 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.334568    4320 generic.go:356] "Generic (PLEG): container finished" podID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerID="43b6a3341af30b48d959d73b444631b350c528ff77260c4898a42773a375084f" exitCode=60
Oct 30 20:00:17.334781 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.334632    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerDied","Data":"43b6a3341af30b48d959d73b444631b350c528ff77260c4898a42773a375084f"}
Oct 30 20:00:17.343036 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.342408    4320 scope.go:122] "RemoveContainer" containerID="0835a211ab25008fbfa908430c0b3a7b5bcf41c39f0dd8ca812089227e343df2"
Oct 30 20:00:17.364306 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.364281    4320 scope.go:122] "RemoveContainer" containerID="fe86a386f4d29f377a34a034d75d60c6b5b7332d0397f1ad2f64de871532b7d3"
Oct 30 20:00:17.365194 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.365119    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:17.418334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.417010    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:17.453066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.452969    4320 scope.go:122] "RemoveContainer" containerID="e42d2a94feb5678b7b6ae8161f6d9a3fe50968fdaeea01d6a422bfd69f9dbb75"
Oct 30 20:00:17.466899 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.466495    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:17.476559 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.476531    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:17.593438 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.593417    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:17.623744 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.623716    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\") pod \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") "
Oct 30 20:00:17.623880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.623803    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\") pod \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\" (UID: \"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba\") "
Oct 30 20:00:17.625530 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.625490    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca" pod "f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" (UID: "f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:17.626241 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.626218    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:17.634682 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.634654    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx" pod "f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" (UID: "f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba"). InnerVolumeSpecName "kube-api-access-2x4lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:17.729565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.729538    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4lx\" (UniqueName: \"kubernetes.io/projected/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba-kube-api-access-2x4lx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:17.749318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.749288    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:17.751367 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.751344    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:17.831414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.831382    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\") pod \"bec81397-7f07-43b8-a73f-acae90decc27\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") "
Oct 30 20:00:17.831582 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.831427    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\") pod \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") "
Oct 30 20:00:17.831687 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.831581    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\" (UniqueName: \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\") pod \"bec81397-7f07-43b8-a73f-acae90decc27\" (UID: \"bec81397-7f07-43b8-a73f-acae90decc27\") "
Oct 30 20:00:17.831687 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.831675    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\" (UniqueName: \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\") pod \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\" (UID: \"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150\") "
Oct 30 20:00:17.834755 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.833237    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca" pod "bec81397-7f07-43b8-a73f-acae90decc27" (UID: "bec81397-7f07-43b8-a73f-acae90decc27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:17.834755 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.834402    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca" pod "5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" (UID: "5d9cfbc6-f03b-4a46-adf5-9da7ab25a150"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:17.835786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.835757    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699" pod "5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" (UID: "5d9cfbc6-f03b-4a46-adf5-9da7ab25a150"). InnerVolumeSpecName "kube-api-access-cg699". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:17.837146 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.837125    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf" pod "bec81397-7f07-43b8-a73f-acae90decc27" (UID: "bec81397-7f07-43b8-a73f-acae90decc27"). InnerVolumeSpecName "kube-api-access-ld8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:17.933449 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.933423    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bec81397-7f07-43b8-a73f-acae90decc27-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:17.933449 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.933448    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:17.933636 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.933457    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld8xf\" (UniqueName: \"kubernetes.io/projected/bec81397-7f07-43b8-a73f-acae90decc27-kube-api-access-ld8xf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:17.933636 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:17.933465    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg699\" (UniqueName: \"kubernetes.io/projected/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150-kube-api-access-cg699\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:18.089539 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.089483    4320 scope.go:122] "RemoveContainer" containerID="27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"
Oct 30 20:00:18.089670 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.089620    4320 scope.go:122] "RemoveContainer" containerID="1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"
Oct 30 20:00:18.089711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.089699    4320 scope.go:122] "RemoveContainer" containerID="8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"
Oct 30 20:00:18.343452 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.343378    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"
Oct 30 20:00:18.343452 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.343428    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6" event={"ID":"f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba","Type":"ContainerDied","Data":"a7a9653307aa6bf3a1fb52f2ac57f437eb71b6d8ee16ff50eb37c747d7f3ae0d"}
Oct 30 20:00:18.343989 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.343486    4320 scope.go:122] "RemoveContainer" containerID="43b6a3341af30b48d959d73b444631b350c528ff77260c4898a42773a375084f"
Oct 30 20:00:18.345709 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.345678    4320 generic.go:356] "Generic (PLEG): container finished" podID="6ef1776d-7576-4554-94e0-30b201c7009c" containerID="8533be43d799639c63e57dd662c1edb26a089136df68a1c36fa0c86619bc605f" exitCode=60
Oct 30 20:00:18.345862 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.345807    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerDied","Data":"8533be43d799639c63e57dd662c1edb26a089136df68a1c36fa0c86619bc605f"}
Oct 30 20:00:18.346213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.346194    4320 scope.go:122] "RemoveContainer" containerID="8533be43d799639c63e57dd662c1edb26a089136df68a1c36fa0c86619bc605f"
Oct 30 20:00:18.346490 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:18.346473    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq_odh-ai-gateway-infra(6ef1776d-7576-4554-94e0-30b201c7009c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" podUID="6ef1776d-7576-4554-94e0-30b201c7009c"
Oct 30 20:00:18.348107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.348087    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"
Oct 30 20:00:18.348211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.348092    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw" event={"ID":"bec81397-7f07-43b8-a73f-acae90decc27","Type":"ContainerDied","Data":"9aac4b28e9eb967add8c914ec7302431f4dac6caa09795ac9959f9e3577c5adf"}
Oct 30 20:00:18.349679 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.349661    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"
Oct 30 20:00:18.349786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.349674    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd" event={"ID":"5d9cfbc6-f03b-4a46-adf5-9da7ab25a150","Type":"ContainerDied","Data":"a703dc5723ff9b03e36fa38b505028c5e41add85680c35df3dba26c6e45692f0"}
Oct 30 20:00:18.355620 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.355583    4320 scope.go:122] "RemoveContainer" containerID="a9a24611d2e3930a804cad1b44ef07951fdad8e460c5224d157a08593b495013"
Oct 30 20:00:18.360295 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.360272    4320 generic.go:356] "Generic (PLEG): container finished" podID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerID="2a06e90342fbe343acfcb712451cf5cf4892a461a4d5729a132f4cdad2ad22f7" exitCode=60
Oct 30 20:00:18.360399 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.360338    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerDied","Data":"2a06e90342fbe343acfcb712451cf5cf4892a461a4d5729a132f4cdad2ad22f7"}
Oct 30 20:00:18.360701 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.360685    4320 scope.go:122] "RemoveContainer" containerID="2a06e90342fbe343acfcb712451cf5cf4892a461a4d5729a132f4cdad2ad22f7"
Oct 30 20:00:18.360949 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:18.360931    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5_odh-ai-gateway-infra(95cdbcb2-e423-4694-8b14-9a0c6ae2e987)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987"
Oct 30 20:00:18.372362 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.372338    4320 scope.go:122] "RemoveContainer" containerID="850060dba9783f5d173494988fdf2d0689c3d1f4e6af7212fbb0a904407045bb"
Oct 30 20:00:18.420057 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.412373    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:18.450051 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.449935    4320 scope.go:122] "RemoveContainer" containerID="1344fdf269b54609fc9c81d0e24a0b7b8409a33a68e95366439a2b691a4f10c7"
Oct 30 20:00:18.467304 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.467273    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-870193-29889840-hdtsw"]
Oct 30 20:00:18.468090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.468007    4320 scope.go:122] "RemoveContainer" containerID="f8153a4795410e2574a83166e20953f6d0b00c632df1607aff14a7a76305c83f"
Oct 30 20:00:18.473639 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.473509    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 20:00:18.473880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.473851    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy" containerID="cri-o://8b4c271fd4f3619100f9ae07b93e2c303a5087d928d21d1f370a3c76d317e6dc" gracePeriod=30
Oct 30 20:00:18.508697 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.508671    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:18.512643 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.512571    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2afc11-29889840-r8mwd"]
Oct 30 20:00:18.618839 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.596846    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:18.770817 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.770776    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:18.787800 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.787759    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b97379-29889840-t9ng6"]
Oct 30 20:00:18.872116 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.872053    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:18.947786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.947486    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:18.955355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.955275    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:18.998143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998111    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"]
Oct 30 20:00:18.998912 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998889    4320 cpu_manager.go:414] "containerMap: removing container" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.998912 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998914    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998937    4320 cpu_manager.go:414] "containerMap: removing container" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998944    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998955    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998963    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998978    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.998984    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999003    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999009    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999018    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999025    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999039    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999045    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999060    4320 cpu_manager.go:414] "containerMap: removing container" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999066    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999098    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999106    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999122    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999129    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999144    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999151    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999169    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999177    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999359    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999373    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999383    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999394    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999407    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999416    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999424    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999435    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999451    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:00:18.999974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:18.999462    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:00:19.008244 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.008224    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.028243 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.028207    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\" (UniqueName: \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\") pod \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") "
Oct 30 20:00:19.028499 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.028482    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\" (UniqueName: \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\") pod \"82404b8d-352d-487b-991f-5931d7c74b1a\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") "
Oct 30 20:00:19.030933 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.030894    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca" pod "82404b8d-352d-487b-991f-5931d7c74b1a" (UID: "82404b8d-352d-487b-991f-5931d7c74b1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.031154 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.030536    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\") pod \"82404b8d-352d-487b-991f-5931d7c74b1a\" (UID: \"82404b8d-352d-487b-991f-5931d7c74b1a\") "
Oct 30 20:00:19.031318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.031295    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\") pod \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\" (UID: \"2371dc5c-ddc4-44f9-a83f-d2753aa58873\") "
Oct 30 20:00:19.033627 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.032312    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"]
Oct 30 20:00:19.033627 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.032515    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:19.033627 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.033177    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca" pod "2371dc5c-ddc4-44f9-a83f-d2753aa58873" (UID: "2371dc5c-ddc4-44f9-a83f-d2753aa58873"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.034980 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.034925    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82404b8d-352d-487b-991f-5931d7c74b1a-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.034980 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.034949    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2371dc5c-ddc4-44f9-a83f-d2753aa58873-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.038673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.038621    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m" pod "2371dc5c-ddc4-44f9-a83f-d2753aa58873" (UID: "2371dc5c-ddc4-44f9-a83f-d2753aa58873"). InnerVolumeSpecName "kube-api-access-8sp9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.068097 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.068064    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt" pod "82404b8d-352d-487b-991f-5931d7c74b1a" (UID: "82404b8d-352d-487b-991f-5931d7c74b1a"). InnerVolumeSpecName "kube-api-access-z28qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.143294 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.143262    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.143624 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.143592    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqdt\" (UniqueName: \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.143744 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.143729    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.143901 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.143890    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sp9m\" (UniqueName: \"kubernetes.io/projected/2371dc5c-ddc4-44f9-a83f-d2753aa58873-kube-api-access-8sp9m\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.143981 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.143971    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z28qt\" (UniqueName: \"kubernetes.io/projected/82404b8d-352d-487b-991f-5931d7c74b1a-kube-api-access-z28qt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.145844 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.145817    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:19.244836 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.244787    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\") pod \"a55e4284-8465-45aa-b169-056a83859ed4\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") "
Oct 30 20:00:19.245023 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.244879    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\" (UniqueName: \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\") pod \"a55e4284-8465-45aa-b169-056a83859ed4\" (UID: \"a55e4284-8465-45aa-b169-056a83859ed4\") "
Oct 30 20:00:19.245237 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.245208    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca" pod "a55e4284-8465-45aa-b169-056a83859ed4" (UID: "a55e4284-8465-45aa-b169-056a83859ed4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.245456 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.245427    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.245637 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.245589    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqdt\" (UniqueName: \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.245733 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.245647    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.245853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.245820    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a55e4284-8465-45aa-b169-056a83859ed4-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.246321 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.246296    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.248024 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.247985    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw" pod "a55e4284-8465-45aa-b169-056a83859ed4" (UID: "a55e4284-8465-45aa-b169-056a83859ed4"). InnerVolumeSpecName "kube-api-access-4gwpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.263784 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.263763    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.267096 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.267074    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqdt\" (UniqueName: \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.337130 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.337102    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:19.346993 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.346967    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gwpw\" (UniqueName: \"kubernetes.io/projected/a55e4284-8465-45aa-b169-056a83859ed4-kube-api-access-4gwpw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.379402 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.379369    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerStarted","Data":"bf24f07e20a989689b58d09ce0d541f9e3fdf1262f968cf490b0b8bb9502f568"}
Oct 30 20:00:19.462621 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.433793    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerStarted","Data":"add36984cfa30671c666980569a175e7a6f5cf1bfc9d5cff1afca29b2d5b23d1"}
Oct 30 20:00:19.560847 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.519318    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr" event={"ID":"82404b8d-352d-487b-991f-5931d7c74b1a","Type":"ContainerDied","Data":"7ef3094612805e4b7bf2317d8d7ee561ce40e334ea69743099f0861cc62ff5c8"}
Oct 30 20:00:19.560847 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.519382    4320 scope.go:122] "RemoveContainer" containerID="6763119322e304194241454a8510cd636a069a48c0a1acd5dd2a6efcd42de4e5"
Oct 30 20:00:19.560847 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.519520    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"
Oct 30 20:00:19.648415 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.632004    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerStarted","Data":"4da44e9b610cadc3417074374a4dbec278193f2eff8169ce73d9cb5c9638a8ad"}
Oct 30 20:00:19.654626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.650736    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"
Oct 30 20:00:19.654626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.651052    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f" event={"ID":"2371dc5c-ddc4-44f9-a83f-d2753aa58873","Type":"ContainerDied","Data":"5ac347f4fb939c6ce5057221e1b44e281e6d4f3a1a8a2afe90165cc91b151732"}
Oct 30 20:00:19.654626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.652808    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:19.654626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.653255    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z" event={"ID":"a55e4284-8465-45aa-b169-056a83859ed4","Type":"ContainerDied","Data":"e762c00dba5fa319bf876ed56bac20f9cbf96f0ef1e15f8ec7ff2e4f8087b92b"}
Oct 30 20:00:19.654626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.653355    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"
Oct 30 20:00:19.657076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.657056    4320 scope.go:122] "RemoveContainer" containerID="76c2eea897a6e0ed1bf71546db7f270e15a540bf492954877d62e129e1e84780"
Oct 30 20:00:19.690783 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.690759    4320 scope.go:122] "RemoveContainer" containerID="6bf02228513481f59a7ece7f687335e412fefb432fdf08f972b325e46afeecb5"
Oct 30 20:00:19.725743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.725713    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:19.737632 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.737555    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-f83282-29889840-jnb8z"]
Oct 30 20:00:19.804721 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.804692    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"]
Oct 30 20:00:19.827945 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.827918    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:19.837987 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.837966    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:19.844991 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.844967    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-ec2fe7-29889840-87rlr"]
Oct 30 20:00:19.884342 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.884304    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\") pod \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") "
Oct 30 20:00:19.884638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.884586    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\" (UniqueName: \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\") pod \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\" (UID: \"95cdbcb2-e423-4694-8b14-9a0c6ae2e987\") "
Oct 30 20:00:19.884741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.884711    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca" pod "95cdbcb2-e423-4694-8b14-9a0c6ae2e987" (UID: "95cdbcb2-e423-4694-8b14-9a0c6ae2e987"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.886674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.886574    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:19.886674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.886630    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.888259 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.888221    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s" pod "95cdbcb2-e423-4694-8b14-9a0c6ae2e987" (UID: "95cdbcb2-e423-4694-8b14-9a0c6ae2e987"). InnerVolumeSpecName "kube-api-access-vmv6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.896183 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.896160    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:19.905574 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.905506    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fe3e5e-29889840-xsf7f"]
Oct 30 20:00:19.994590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.989089    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\") pod \"6ef1776d-7576-4554-94e0-30b201c7009c\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") "
Oct 30 20:00:19.994590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.989231    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\" (UniqueName: \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\") pod \"6ef1776d-7576-4554-94e0-30b201c7009c\" (UID: \"6ef1776d-7576-4554-94e0-30b201c7009c\") "
Oct 30 20:00:19.994590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.990085    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmv6s\" (UniqueName: \"kubernetes.io/projected/95cdbcb2-e423-4694-8b14-9a0c6ae2e987-kube-api-access-vmv6s\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.994590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.990802    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca" pod "6ef1776d-7576-4554-94e0-30b201c7009c" (UID: "6ef1776d-7576-4554-94e0-30b201c7009c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.994590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:19.993883    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw" pod "6ef1776d-7576-4554-94e0-30b201c7009c" (UID: "6ef1776d-7576-4554-94e0-30b201c7009c"). InnerVolumeSpecName "kube-api-access-997nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:20.090957 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.090929    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1776d-7576-4554-94e0-30b201c7009c-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.090957 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.090958    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997nw\" (UniqueName: \"kubernetes.io/projected/6ef1776d-7576-4554-94e0-30b201c7009c-kube-api-access-997nw\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:20.099036 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.099002    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" path="/var/lib/kubelet/pods/2371dc5c-ddc4-44f9-a83f-d2753aa58873/volumes"
Oct 30 20:00:20.099456 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.099442    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" path="/var/lib/kubelet/pods/5d9cfbc6-f03b-4a46-adf5-9da7ab25a150/volumes"
Oct 30 20:00:20.099911 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.099895    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" path="/var/lib/kubelet/pods/82404b8d-352d-487b-991f-5931d7c74b1a/volumes"
Oct 30 20:00:20.100582 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.100565    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55e4284-8465-45aa-b169-056a83859ed4" path="/var/lib/kubelet/pods/a55e4284-8465-45aa-b169-056a83859ed4/volumes"
Oct 30 20:00:20.100993 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.100979    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec81397-7f07-43b8-a73f-acae90decc27" path="/var/lib/kubelet/pods/bec81397-7f07-43b8-a73f-acae90decc27/volumes"
Oct 30 20:00:20.101373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.101359    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" path="/var/lib/kubelet/pods/f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba/volumes"
Oct 30 20:00:20.509523 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.509443    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:20.551578 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.551549    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:20.663865 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.663831    4320 generic.go:356] "Generic (PLEG): container finished" podID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerID="4da44e9b610cadc3417074374a4dbec278193f2eff8169ce73d9cb5c9638a8ad" exitCode=60
Oct 30 20:00:20.664039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.663899    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerDied","Data":"4da44e9b610cadc3417074374a4dbec278193f2eff8169ce73d9cb5c9638a8ad"}
Oct 30 20:00:20.664039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.663961    4320 scope.go:122] "RemoveContainer" containerID="8e2e337f6ed1461db8bf9c803fa9153e9c0e6076e98d13c8795e8f3150aaf089"
Oct 30 20:00:20.664380 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.664356    4320 scope.go:122] "RemoveContainer" containerID="4da44e9b610cadc3417074374a4dbec278193f2eff8169ce73d9cb5c9638a8ad"
Oct 30 20:00:20.664719 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:20.664694    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh_odh-ai-gateway-infra(37ebcd68-f7db-4238-87db-38b0c1b83d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a"
Oct 30 20:00:20.667442 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.667425    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"
Oct 30 20:00:20.667541 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.667467    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5" event={"ID":"95cdbcb2-e423-4694-8b14-9a0c6ae2e987","Type":"ContainerDied","Data":"cad02ba0620b4727ba0be0bbf4d8262032bb71ce26f9f306203b305d070deb7c"}
Oct 30 20:00:20.669662 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.669635    4320 generic.go:356] "Generic (PLEG): container finished" podID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerID="bf24f07e20a989689b58d09ce0d541f9e3fdf1262f968cf490b0b8bb9502f568" exitCode=60
Oct 30 20:00:20.669763 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.669741    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerDied","Data":"bf24f07e20a989689b58d09ce0d541f9e3fdf1262f968cf490b0b8bb9502f568"}
Oct 30 20:00:20.672011 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.671983    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6" event={"ID":"59e8727b-6ad8-4555-9257-3235e7e9375a","Type":"ContainerStarted","Data":"c809092193d49a86d4378dd0861824cb6464926303d7d3d3899fd9a6013df5c6"}
Oct 30 20:00:20.672118 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.672035    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6" event={"ID":"59e8727b-6ad8-4555-9257-3235e7e9375a","Type":"ContainerStarted","Data":"50385ede707ac610eb722dc0d2d30acf1e20b6061feb87c9091fdc63b9b4cc4e"}
Oct 30 20:00:20.674151 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.674123    4320 generic.go:356] "Generic (PLEG): container finished" podID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerID="add36984cfa30671c666980569a175e7a6f5cf1bfc9d5cff1afca29b2d5b23d1" exitCode=60
Oct 30 20:00:20.674257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.674215    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerDied","Data":"add36984cfa30671c666980569a175e7a6f5cf1bfc9d5cff1afca29b2d5b23d1"}
Oct 30 20:00:20.677004 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.676978    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"
Oct 30 20:00:20.677004 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.676986    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq" event={"ID":"6ef1776d-7576-4554-94e0-30b201c7009c","Type":"ContainerDied","Data":"72fe363744fdc7cdc26a33d9dfd820e6441b02f6df27179d7d050c9cb8cb9082"}
Oct 30 20:00:20.678415 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.678388    4320 scope.go:122] "RemoveContainer" containerID="2a06e90342fbe343acfcb712451cf5cf4892a461a4d5729a132f4cdad2ad22f7"
Oct 30 20:00:20.688541 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.688521    4320 scope.go:122] "RemoveContainer" containerID="1b7c6334e2279ce93c99c483b8889b344ea0e4f310f6184009e4c27c9c29e86e"
Oct 30 20:00:20.714077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.709675    4320 scope.go:122] "RemoveContainer" containerID="27848de397d0f4084e92013b74c7bdb995bf4cefa1ef695e450bedd84c89f7eb"
Oct 30 20:00:20.714077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.711926    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:20.736432 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.736406    4320 scope.go:122] "RemoveContainer" containerID="8533be43d799639c63e57dd662c1edb26a089136df68a1c36fa0c86619bc605f"
Oct 30 20:00:20.790395 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.790359    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:20.795621 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.795578    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf314f-29889840-p6rn5"]
Oct 30 20:00:20.833883 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.833826    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6" podStartSLOduration=2.83380789 podStartE2EDuration="2.83380789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.830099912 +0000 UTC m=+3368.865612376" watchObservedRunningTime="2026-10-30 20:00:20.83380789 +0000 UTC m=+3368.869320357"
Oct 30 20:00:20.926686 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.926661    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:20.929724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.929707    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:20.953344 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.953321    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:20.958467 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:20.958446    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-095f55-29889840-jdczq"]
Oct 30 20:00:21.002942 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.002913    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\") pod \"f881ead9-5adb-443c-aabb-8792af9e5cca\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") "
Oct 30 20:00:21.003082 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.003064    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\" (UniqueName: \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\") pod \"f881ead9-5adb-443c-aabb-8792af9e5cca\" (UID: \"f881ead9-5adb-443c-aabb-8792af9e5cca\") "
Oct 30 20:00:21.003194 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.003176    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\" (UniqueName: \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\") pod \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") "
Oct 30 20:00:21.003298 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.003283    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\") pod \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\" (UID: \"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae\") "
Oct 30 20:00:21.004371 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.004341    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca" pod "c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" (UID: "c2b259aa-f4d5-46c0-be7e-3b95c53e13ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.004741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.004710    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca" pod "f881ead9-5adb-443c-aabb-8792af9e5cca" (UID: "f881ead9-5adb-443c-aabb-8792af9e5cca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.007304 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.007272    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z" pod "f881ead9-5adb-443c-aabb-8792af9e5cca" (UID: "f881ead9-5adb-443c-aabb-8792af9e5cca"). InnerVolumeSpecName "kube-api-access-v2s9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.007725 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.007700    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq" pod "c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" (UID: "c2b259aa-f4d5-46c0-be7e-3b95c53e13ae"). InnerVolumeSpecName "kube-api-access-2xdvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.105461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.105321    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.105461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.105351    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f881ead9-5adb-443c-aabb-8792af9e5cca-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.105461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.105368    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2s9z\" (UniqueName: \"kubernetes.io/projected/f881ead9-5adb-443c-aabb-8792af9e5cca-kube-api-access-v2s9z\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.105461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.105384    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdvq\" (UniqueName: \"kubernetes.io/projected/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae-kube-api-access-2xdvq\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:21.687641 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.687613    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"
Oct 30 20:00:21.688020 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.687645    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz" event={"ID":"c2b259aa-f4d5-46c0-be7e-3b95c53e13ae","Type":"ContainerDied","Data":"8431c5e40bf87d59daec8faf9d3bf564ca9cb3f24ffd3d0986594143d7679592"}
Oct 30 20:00:21.688020 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.687700    4320 scope.go:122] "RemoveContainer" containerID="bf24f07e20a989689b58d09ce0d541f9e3fdf1262f968cf490b0b8bb9502f568"
Oct 30 20:00:21.689409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.689371    4320 generic.go:356] "Generic (PLEG): container finished" podID="59e8727b-6ad8-4555-9257-3235e7e9375a" containerID="c809092193d49a86d4378dd0861824cb6464926303d7d3d3899fd9a6013df5c6" exitCode=0
Oct 30 20:00:21.689526 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.689480    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6" event={"ID":"59e8727b-6ad8-4555-9257-3235e7e9375a","Type":"ContainerDied","Data":"c809092193d49a86d4378dd0861824cb6464926303d7d3d3899fd9a6013df5c6"}
Oct 30 20:00:21.691141 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.691118    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"
Oct 30 20:00:21.691222 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.691134    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx" event={"ID":"f881ead9-5adb-443c-aabb-8792af9e5cca","Type":"ContainerDied","Data":"2094bc3a48f37db0501d14c5998596e7f0ba34be4e857be6d95aaf2243c29709"}
Oct 30 20:00:21.700116 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.700096    4320 scope.go:122] "RemoveContainer" containerID="add36984cfa30671c666980569a175e7a6f5cf1bfc9d5cff1afca29b2d5b23d1"
Oct 30 20:00:21.842293 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.842268    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:21.843978 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.843956    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:21.862208 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.862185    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b7ccc-29889840-x8mwz"]
Oct 30 20:00:21.914461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.914430    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\" (UniqueName: \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\") pod \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") "
Oct 30 20:00:21.914586 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.914473    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\") pod \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\" (UID: \"37ebcd68-f7db-4238-87db-38b0c1b83d3a\") "
Oct 30 20:00:21.914880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.914856    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca" pod "37ebcd68-f7db-4238-87db-38b0c1b83d3a" (UID: "37ebcd68-f7db-4238-87db-38b0c1b83d3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.916391 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.916369    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7" pod "37ebcd68-f7db-4238-87db-38b0c1b83d3a" (UID: "37ebcd68-f7db-4238-87db-38b0c1b83d3a"). InnerVolumeSpecName "kube-api-access-64ss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.943637 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.943564    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:21.968307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:21.968281    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c5449c-29889840-9hzpx"]
Oct 30 20:00:22.016033 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.016010    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ebcd68-f7db-4238-87db-38b0c1b83d3a-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.016033 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.016034    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ss7\" (UniqueName: \"kubernetes.io/projected/37ebcd68-f7db-4238-87db-38b0c1b83d3a-kube-api-access-64ss7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:22.099585 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.099543    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" path="/var/lib/kubelet/pods/6ef1776d-7576-4554-94e0-30b201c7009c/volumes"
Oct 30 20:00:22.100115 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.100092    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" path="/var/lib/kubelet/pods/95cdbcb2-e423-4694-8b14-9a0c6ae2e987/volumes"
Oct 30 20:00:22.100586 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.100565    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" path="/var/lib/kubelet/pods/c2b259aa-f4d5-46c0-be7e-3b95c53e13ae/volumes"
Oct 30 20:00:22.101369 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.101346    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" path="/var/lib/kubelet/pods/f881ead9-5adb-443c-aabb-8792af9e5cca/volumes"
Oct 30 20:00:22.708858 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.708832    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"
Oct 30 20:00:22.709302 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.708861    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh" event={"ID":"37ebcd68-f7db-4238-87db-38b0c1b83d3a","Type":"ContainerDied","Data":"f0063fc801911c0ac065abea76a0058d4b3f0f15610113a063624d3ee35abf3d"}
Oct 30 20:00:22.709302 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.708927    4320 scope.go:122] "RemoveContainer" containerID="4da44e9b610cadc3417074374a4dbec278193f2eff8169ce73d9cb5c9638a8ad"
Oct 30 20:00:22.766452 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.766422    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:22.770571 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.770494    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e67b84-29889840-zmcjh"]
Oct 30 20:00:22.927245 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:22.927223    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:23.029533 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.029466    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\" (UniqueName: \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\") pod \"59e8727b-6ad8-4555-9257-3235e7e9375a\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") "
Oct 30 20:00:23.029700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.029542    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\") pod \"59e8727b-6ad8-4555-9257-3235e7e9375a\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") "
Oct 30 20:00:23.029700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.029647    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\") pod \"59e8727b-6ad8-4555-9257-3235e7e9375a\" (UID: \"59e8727b-6ad8-4555-9257-3235e7e9375a\") "
Oct 30 20:00:23.030052 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.030025    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca" pod "59e8727b-6ad8-4555-9257-3235e7e9375a" (UID: "59e8727b-6ad8-4555-9257-3235e7e9375a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.030226 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.030208    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.031647 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.031624    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca" pod "59e8727b-6ad8-4555-9257-3235e7e9375a" (UID: "59e8727b-6ad8-4555-9257-3235e7e9375a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:23.031757 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.031714    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt" pod "59e8727b-6ad8-4555-9257-3235e7e9375a" (UID: "59e8727b-6ad8-4555-9257-3235e7e9375a"). InnerVolumeSpecName "kube-api-access-rdqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.131777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.131749    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e8727b-6ad8-4555-9257-3235e7e9375a-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.131777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.131773    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqdt\" (UniqueName: \"kubernetes.io/projected/59e8727b-6ad8-4555-9257-3235e7e9375a-kube-api-access-rdqdt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.722211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.722182    4320 generic.go:356] "Generic (PLEG): container finished" podID="09713a4d-4542-4978-a9e8-9f2915794d50" containerID="8b4c271fd4f3619100f9ae07b93e2c303a5087d928d21d1f370a3c76d317e6dc" exitCode=0
Oct 30 20:00:23.722630 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.722268    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerDied","Data":"8b4c271fd4f3619100f9ae07b93e2c303a5087d928d21d1f370a3c76d317e6dc"}
Oct 30 20:00:23.722630 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.722343    4320 scope.go:122] "RemoveContainer" containerID="ec7d9966b1f335ec06da68d58e7ca0f17ca166ad613ccb31087eaf10b44d3c36"
Oct 30 20:00:23.723799 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.723774    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6" event={"ID":"59e8727b-6ad8-4555-9257-3235e7e9375a","Type":"ContainerDied","Data":"50385ede707ac610eb722dc0d2d30acf1e20b6061feb87c9091fdc63b9b4cc4e"}
Oct 30 20:00:23.723904 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.723800    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"
Oct 30 20:00:23.723904 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.723822    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50385ede707ac610eb722dc0d2d30acf1e20b6061feb87c9091fdc63b9b4cc4e"
Oct 30 20:00:23.731538 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.731506    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 20:00:23.841266 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841196    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841347    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841379    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841500 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841434    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841562 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841505    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841635 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841575    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.841635 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841616    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841747 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841638    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841747 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841707    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841864 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841759    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\") pod \"09713a4d-4542-4978-a9e8-9f2915794d50\" (UID: \"09713a4d-4542-4978-a9e8-9f2915794d50\") "
Oct 30 20:00:23.841864 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841818    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:23.841978 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.841955    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:23.842073 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842055    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:23.842137 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842124    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:23.842415 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842397    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.842487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842420    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.842487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842434    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.842487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842446    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.842487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.842459    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09713a4d-4542-4978-a9e8-9f2915794d50-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.843516 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.843496    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:23.844018 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.843999    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:23.844083 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.844035    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "kube-api-access-kdxfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.844251 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.844234    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token" pod "09713a4d-4542-4978-a9e8-9f2915794d50" (UID: "09713a4d-4542-4978-a9e8-9f2915794d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.943243 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.943217    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdxfb\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-kube-api-access-kdxfb\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.943243 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.943242    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09713a4d-4542-4978-a9e8-9f2915794d50-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.943384 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.943250    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09713a4d-4542-4978-a9e8-9f2915794d50-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:23.943384 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:23.943258    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09713a4d-4542-4978-a9e8-9f2915794d50-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:24.098054 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.097968    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" path="/var/lib/kubelet/pods/37ebcd68-f7db-4238-87db-38b0c1b83d3a/volumes"
Oct 30 20:00:24.731988 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.731960    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"
Oct 30 20:00:24.732465 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.731986    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6" event={"ID":"09713a4d-4542-4978-a9e8-9f2915794d50","Type":"ContainerDied","Data":"96165eb459753dba46bd803cd4c90aa04d0ce259f4715d8323a235edecd418b3"}
Oct 30 20:00:24.732465 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.732048    4320 scope.go:122] "RemoveContainer" containerID="8b4c271fd4f3619100f9ae07b93e2c303a5087d928d21d1f370a3c76d317e6dc"
Oct 30 20:00:24.823981 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.823927    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 20:00:24.828458 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:24.828436    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a56cb5-openshift-default-74dd666b9c-qmvm6"]
Oct 30 20:00:26.098232 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:26.098202    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" path="/var/lib/kubelet/pods/09713a4d-4542-4978-a9e8-9f2915794d50/volumes"
Oct 30 20:00:26.733369 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:26.733330    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 20:00:26.735171 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:26.734909    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy" containerID="cri-o://8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4" gracePeriod=30
Oct 30 20:00:26.739161 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:26.739134    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 20:00:26.739376 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:26.739348    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy" containerID="cri-o://d668053e13e8d5303914a51dc84f7f6d784e0f43bef589f53096d59dbc8137b7" gracePeriod=30
Oct 30 20:00:28.650253 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:28.650217    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:00:30.261741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.261704    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 20:00:30.262253 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.261976    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" podUID="198b2307-c817-4ed3-b88e-7d07b3383357" containerName="maas-api" containerID="cri-o://e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9" gracePeriod=30
Oct 30 20:00:30.591415 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.591386    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 20:00:30.647676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.647647    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\" (UniqueName: \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\") pod \"198b2307-c817-4ed3-b88e-7d07b3383357\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") "
Oct 30 20:00:30.647676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.647686    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\") pod \"198b2307-c817-4ed3-b88e-7d07b3383357\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") "
Oct 30 20:00:30.647936 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.647848    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\") pod \"198b2307-c817-4ed3-b88e-7d07b3383357\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") "
Oct 30 20:00:30.647999 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.647982    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\") pod \"198b2307-c817-4ed3-b88e-7d07b3383357\" (UID: \"198b2307-c817-4ed3-b88e-7d07b3383357\") "
Oct 30 20:00:30.648760 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.648717    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca" pod "198b2307-c817-4ed3-b88e-7d07b3383357" (UID: "198b2307-c817-4ed3-b88e-7d07b3383357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:30.650667 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.650634    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls" pod "198b2307-c817-4ed3-b88e-7d07b3383357" (UID: "198b2307-c817-4ed3-b88e-7d07b3383357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:30.650989 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.650962    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs" pod "198b2307-c817-4ed3-b88e-7d07b3383357" (UID: "198b2307-c817-4ed3-b88e-7d07b3383357"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:30.652640 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.652122    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7" pod "198b2307-c817-4ed3-b88e-7d07b3383357" (UID: "198b2307-c817-4ed3-b88e-7d07b3383357"). InnerVolumeSpecName "kube-api-access-d9cn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.750421 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.750382    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/198b2307-c817-4ed3-b88e-7d07b3383357-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.750421 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.750409    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9cn7\" (UniqueName: \"kubernetes.io/projected/198b2307-c817-4ed3-b88e-7d07b3383357-kube-api-access-d9cn7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.750421 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.750419    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.750421 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.750428    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/198b2307-c817-4ed3-b88e-7d07b3383357-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.802012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.801974    4320 generic.go:356] "Generic (PLEG): container finished" podID="198b2307-c817-4ed3-b88e-7d07b3383357" containerID="e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9" exitCode=0
Oct 30 20:00:30.802012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.802033    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"
Oct 30 20:00:30.802012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.802066    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" event={"ID":"198b2307-c817-4ed3-b88e-7d07b3383357","Type":"ContainerDied","Data":"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"}
Oct 30 20:00:30.802012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.802159    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj" event={"ID":"198b2307-c817-4ed3-b88e-7d07b3383357","Type":"ContainerDied","Data":"a030ec5caf97702757969684770524693d7bdaeec738b445a827f21310469167"}
Oct 30 20:00:30.802012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.802215    4320 scope.go:122] "RemoveContainer" containerID="e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"
Oct 30 20:00:30.818623 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.818564    4320 scope.go:122] "RemoveContainer" containerID="e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"
Oct 30 20:00:30.818930 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:30.818911    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9\": container with ID starting with e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9 not found: ID does not exist" containerID="e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"
Oct 30 20:00:30.818997 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.818938    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9"} err="failed to get container status \"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9\": rpc error: code = NotFound desc = could not find container \"e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9\": container with ID starting with e07467c4438324bc520fa908b79948c1df76690355102d1e77af5a92cb3a76c9 not found: ID does not exist"
Oct 30 20:00:30.857711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.857683    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 20:00:30.861138 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:30.861119    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed2ede-7b6ccddf66-kc6kj"]
Oct 30 20:00:31.708821 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.708784    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"]
Oct 30 20:00:31.711786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.711763    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-f83282-wsfb6"]
Oct 30 20:00:31.811515 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.811488    4320 generic.go:356] "Generic (PLEG): container finished" podID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerID="d668053e13e8d5303914a51dc84f7f6d784e0f43bef589f53096d59dbc8137b7" exitCode=0
Oct 30 20:00:31.811642 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.811532    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" event={"ID":"314e2376-43ec-4006-a31e-5d426ebbf9a2","Type":"ContainerDied","Data":"d668053e13e8d5303914a51dc84f7f6d784e0f43bef589f53096d59dbc8137b7"}
Oct 30 20:00:31.992088 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.992065    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 20:00:31.996034 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.995990    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996161 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996071    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996161 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996153    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996302 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996184    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996302 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996242    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996474 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996441    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996472    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996263    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:31.996615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996503    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996902 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996639    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\") pod \"314e2376-43ec-4006-a31e-5d426ebbf9a2\" (UID: \"314e2376-43ec-4006-a31e-5d426ebbf9a2\") "
Oct 30 20:00:31.996902 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996486    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:31.996902 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996678    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:31.996902 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.996803    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:31.997292 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.997265    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:31.997409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.997361    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.997409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.997386    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.997409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.997402    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.997645 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.997418    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.999823 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.999629    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:31.999823 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.999788    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:31.999999 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.999830    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "kube-api-access-czhzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:31.999999 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:31.999893    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo" pod "314e2376-43ec-4006-a31e-5d426ebbf9a2" (UID: "314e2376-43ec-4006-a31e-5d426ebbf9a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:32.024156 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.024135    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098311    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098371    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098445    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098523    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098556    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098668    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.098778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.098706    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.099318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.099145    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.099447    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.099611    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\") pod \"99864e99-111a-44e4-bd64-8be3c2ec3387\" (UID: \"99864e99-111a-44e4-bd64-8be3c2ec3387\") "
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.099792    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100067    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100291    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100407    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100425    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100444    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99864e99-111a-44e4-bd64-8be3c2ec3387-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100460    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100359    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100474    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314e2376-43ec-4006-a31e-5d426ebbf9a2-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100398    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100489    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhzd\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-kube-api-access-czhzd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.101002 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.100502    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314e2376-43ec-4006-a31e-5d426ebbf9a2-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.102024 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.101996    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:32.103675 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.103648    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198b2307-c817-4ed3-b88e-7d07b3383357" path="/var/lib/kubelet/pods/198b2307-c817-4ed3-b88e-7d07b3383357/volumes"
Oct 30 20:00:32.103996 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.103906    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:32.103996 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.103969    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:32.104347 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.104324    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e8727b-6ad8-4555-9257-3235e7e9375a" path="/var/lib/kubelet/pods/59e8727b-6ad8-4555-9257-3235e7e9375a/volumes"
Oct 30 20:00:32.105235 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.105210    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl" pod "99864e99-111a-44e4-bd64-8be3c2ec3387" (UID: "99864e99-111a-44e4-bd64-8be3c2ec3387"). InnerVolumeSpecName "kube-api-access-rsxgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:32.202066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202028    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.202066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202054    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.202066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202063    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.202066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202071    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.202397 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202079    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99864e99-111a-44e4-bd64-8be3c2ec3387-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.202397 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.202087    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxgl\" (UniqueName: \"kubernetes.io/projected/99864e99-111a-44e4-bd64-8be3c2ec3387-kube-api-access-rsxgl\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:32.819982 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.819950    4320 generic.go:356] "Generic (PLEG): container finished" podID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerID="8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4" exitCode=0
Oct 30 20:00:32.820385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.820004    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"
Oct 30 20:00:32.820385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.820030    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" event={"ID":"99864e99-111a-44e4-bd64-8be3c2ec3387","Type":"ContainerDied","Data":"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"}
Oct 30 20:00:32.820385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.820077    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4" event={"ID":"99864e99-111a-44e4-bd64-8be3c2ec3387","Type":"ContainerDied","Data":"b34ff8783d460fb02fe81423f5bb837383ab24362270d83755fe6eb7d35d8e82"}
Oct 30 20:00:32.820385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.820088    4320 scope.go:122] "RemoveContainer" containerID="8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"
Oct 30 20:00:32.821764 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.821739    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt" event={"ID":"314e2376-43ec-4006-a31e-5d426ebbf9a2","Type":"ContainerDied","Data":"613b5170a2e34284d1929246fe9040a0ee24a19cc43f03abe8ab46b84827c3a5"}
Oct 30 20:00:32.821885 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.821746    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"
Oct 30 20:00:32.837873 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.837850    4320 scope.go:122] "RemoveContainer" containerID="8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"
Oct 30 20:00:32.838349 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:32.838315    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4\": container with ID starting with 8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4 not found: ID does not exist" containerID="8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"
Oct 30 20:00:32.838403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.838361    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4"} err="failed to get container status \"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4\": rpc error: code = NotFound desc = could not find container \"8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4\": container with ID starting with 8f2e4193e1da7817d38f4d5e55f9156ca1bb7811a8d57903651c066ca215a6c4 not found: ID does not exist"
Oct 30 20:00:32.838403 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.838386    4320 scope.go:122] "RemoveContainer" containerID="d668053e13e8d5303914a51dc84f7f6d784e0f43bef589f53096d59dbc8137b7"
Oct 30 20:00:32.884151 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.884126    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 20:00:32.895238 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.895211    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8x4pt"]
Oct 30 20:00:32.920667 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.920649    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 20:00:32.926073 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:32.926052    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-vnfl4"]
Oct 30 20:00:34.096767 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:34.096735    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" path="/var/lib/kubelet/pods/314e2376-43ec-4006-a31e-5d426ebbf9a2/volumes"
Oct 30 20:00:34.097229 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:34.097214    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" path="/var/lib/kubelet/pods/99864e99-111a-44e4-bd64-8be3c2ec3387/volumes"
Oct 30 20:00:44.022233 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:44.022196    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 20:00:44.034229 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:44.034202    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 20:00:44.052746 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:44.052700    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 20:00:58.644734 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:00:58.644693    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:00:59.044695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.044593    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" podUID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" containerName="main" containerID="cri-o://9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374" gracePeriod=45
Oct 30 20:00:59.061084 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.061059    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" containerID="cri-o://32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199" gracePeriod=45
Oct 30 20:00:59.082162 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.082128    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main" containerID="cri-o://0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e" gracePeriod=45
Oct 30 20:00:59.401413 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.401390    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 20:00:59.412627 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.412570    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 20:00:59.414776 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.414752    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 20:00:59.503177 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503144    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.503360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503199    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\" (UniqueName: \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.503360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503234    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.503360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503284    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.503360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503323    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.503609 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503363    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\" (UniqueName: \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.503609 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503417    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.503609 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503447    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.503609 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503526    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\" (UniqueName: \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.503609 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503574    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.503909 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503625    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\" (UniqueName: \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.503909 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503749    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.504017 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503968    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.504017 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.503982    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.504196 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504174    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.504289 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504194    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.504358 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504292    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\") pod \"54fab2ed-de59-4829-88f8-547e0c87e610\" (UID: \"54fab2ed-de59-4829-88f8-547e0c87e610\") "
Oct 30 20:00:59.504418 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504402    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.504474 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504443    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.504544 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504488    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.504544 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504520    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.504690 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504561    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\" (UniqueName: \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\") pod \"cac09df1-8a18-487c-a762-7d47949ff198\" (UID: \"cac09df1-8a18-487c-a762-7d47949ff198\") "
Oct 30 20:00:59.504690 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.504626    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\") pod \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\" (UID: \"30587b0d-6cad-4690-8daa-03b6f2ccbfc5\") "
Oct 30 20:00:59.505348 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505321    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.505472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505351    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.505472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505366    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.505472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505379    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.505472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505393    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.505742 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505655    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.505860 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.505837    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.506493 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.506466    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "kube-api-access-hp69h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:59.506720 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.506702    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.506821 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.506768    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.506905 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.506886    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.509187 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.509159    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "kube-api-access-s54vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:59.509317 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.509223    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:59.509317 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.509293    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm" pod "30587b0d-6cad-4690-8daa-03b6f2ccbfc5" (UID: "30587b0d-6cad-4690-8daa-03b6f2ccbfc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.509446 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.509327    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "kube-api-access-f2vpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:59.509446 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.509360    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs" pod "54fab2ed-de59-4829-88f8-547e0c87e610" (UID: "54fab2ed-de59-4829-88f8-547e0c87e610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:59.516414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.510440    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:59.516414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.511117    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs" pod "cac09df1-8a18-487c-a762-7d47949ff198" (UID: "cac09df1-8a18-487c-a762-7d47949ff198"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.606972    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607001    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac09df1-8a18-487c-a762-7d47949ff198-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607010    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607019    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607027    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac09df1-8a18-487c-a762-7d47949ff198-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607035    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607042    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607046 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607050    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fab2ed-de59-4829-88f8-547e0c87e610-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607058    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54vq\" (UniqueName: \"kubernetes.io/projected/54fab2ed-de59-4829-88f8-547e0c87e610-kube-api-access-s54vq\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607068    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp69h\" (UniqueName: \"kubernetes.io/projected/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-kube-api-access-hp69h\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607078    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30587b0d-6cad-4690-8daa-03b6f2ccbfc5-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607088    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vpd\" (UniqueName: \"kubernetes.io/projected/cac09df1-8a18-487c-a762-7d47949ff198-kube-api-access-f2vpd\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:00:59.607406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:00:59.607097    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fab2ed-de59-4829-88f8-547e0c87e610-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:00.021385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.021348    4320 generic.go:356] "Generic (PLEG): container finished" podID="cac09df1-8a18-487c-a762-7d47949ff198" containerID="32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199" exitCode=0
Oct 30 20:01:00.021865 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.021397    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"
Oct 30 20:01:00.021865 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.021427    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" event={"ID":"cac09df1-8a18-487c-a762-7d47949ff198","Type":"ContainerDied","Data":"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"}
Oct 30 20:01:00.021865 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.021467    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" event={"ID":"cac09df1-8a18-487c-a762-7d47949ff198","Type":"ContainerDied","Data":"93aeffd87e676324acdddde5131734626ec91fa4a5927d7b40eae6e1dfe845f5"}
Oct 30 20:01:00.021865 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.021486    4320 scope.go:122] "RemoveContainer" containerID="32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"
Oct 30 20:01:00.023365 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.023338    4320 generic.go:356] "Generic (PLEG): container finished" podID="54fab2ed-de59-4829-88f8-547e0c87e610" containerID="0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e" exitCode=0
Oct 30 20:01:00.023481 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.023378    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"
Oct 30 20:01:00.023481 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.023416    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" event={"ID":"54fab2ed-de59-4829-88f8-547e0c87e610","Type":"ContainerDied","Data":"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"}
Oct 30 20:01:00.023481 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.023449    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs" event={"ID":"54fab2ed-de59-4829-88f8-547e0c87e610","Type":"ContainerDied","Data":"c764dd34c58d90cee565db73e98eb7071ae81ee2f92bbe38d5f6c284c9f37307"}
Oct 30 20:01:00.025304 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.025280    4320 generic.go:356] "Generic (PLEG): container finished" podID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" containerID="9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374" exitCode=0
Oct 30 20:01:00.025428 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.025320    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" event={"ID":"30587b0d-6cad-4690-8daa-03b6f2ccbfc5","Type":"ContainerDied","Data":"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"}
Oct 30 20:01:00.025428 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.025349    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m" event={"ID":"30587b0d-6cad-4690-8daa-03b6f2ccbfc5","Type":"ContainerDied","Data":"3b020ced5feb04ca5604e7cd98892a920ffd837a3e6de3539f0e5f760b3de334"}
Oct 30 20:01:00.025428 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.025324    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"
Oct 30 20:01:00.033424 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.033408    4320 scope.go:122] "RemoveContainer" containerID="32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"
Oct 30 20:01:00.033758 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:00.033736    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199\": container with ID starting with 32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199 not found: ID does not exist" containerID="32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"
Oct 30 20:01:00.033855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.033762    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199"} err="failed to get container status \"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199\": rpc error: code = NotFound desc = could not find container \"32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199\": container with ID starting with 32d79635d529e15853ccb57808f42dde761f8bb15f661c28e846a152aa331199 not found: ID does not exist"
Oct 30 20:01:00.033855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.033777    4320 scope.go:122] "RemoveContainer" containerID="0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"
Oct 30 20:01:00.042948 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.042934    4320 scope.go:122] "RemoveContainer" containerID="0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"
Oct 30 20:01:00.043178 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:00.043162    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e\": container with ID starting with 0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e not found: ID does not exist" containerID="0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"
Oct 30 20:01:00.043227 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.043186    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e"} err="failed to get container status \"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e\": rpc error: code = NotFound desc = could not find container \"0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e\": container with ID starting with 0ded5dc74b40906242cd397abb67af8d95f6ffa6f42e58837f405caf7af5230e not found: ID does not exist"
Oct 30 20:01:00.043227 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.043200    4320 scope.go:122] "RemoveContainer" containerID="9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"
Oct 30 20:01:00.051748 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.051555    4320 scope.go:122] "RemoveContainer" containerID="9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"
Oct 30 20:01:00.052711 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:00.052686    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374\": container with ID starting with 9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374 not found: ID does not exist" containerID="9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"
Oct 30 20:01:00.052815 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.052723    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374"} err="failed to get container status \"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374\": rpc error: code = NotFound desc = could not find container \"9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374\": container with ID starting with 9809ecb845efd9b0b130acba27d5d13a509839686117783c79dd39aac8236374 not found: ID does not exist"
Oct 30 20:01:00.072312 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.072290    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 20:01:00.084425 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.076036    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x"]
Oct 30 20:01:00.098900 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.098873    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac09df1-8a18-487c-a762-7d47949ff198" path="/var/lib/kubelet/pods/cac09df1-8a18-487c-a762-7d47949ff198/volumes"
Oct 30 20:01:00.154241 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.154205    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 20:01:00.155058 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.154791    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/simulator-w1-5ac614-kserve-b4cf59d8b-2gqxs"]
Oct 30 20:01:00.183877 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.183850    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 20:01:00.187792 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.187765    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/distinct-2-w1-5ac614-kserve-666c989d65-mdn4m"]
Oct 30 20:01:00.293412 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:00.293333    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/distinct-w1-5ac614-kserve-7d674cb4b6-pzp7x" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:02.096706 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:02.096672    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" path="/var/lib/kubelet/pods/30587b0d-6cad-4690-8daa-03b6f2ccbfc5/volumes"
Oct 30 20:01:02.097175 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:02.097159    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" path="/var/lib/kubelet/pods/54fab2ed-de59-4829-88f8-547e0c87e610/volumes"
Oct 30 20:01:02.997248 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:02.997215    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:01:20.642623 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:20.637993    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 20:01:25.639009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:25.638972    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" podUID="c7621967-ab48-4781-bda0-660f356dc505" containerName="payload-processing" containerID="cri-o://34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702" gracePeriod=25
Oct 30 20:01:25.911559 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:25.911496    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 20:01:26.050272 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.050232    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\" (UniqueName: \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\") pod \"c7621967-ab48-4781-bda0-660f356dc505\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") "
Oct 30 20:01:26.050462 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.050419    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\") pod \"c7621967-ab48-4781-bda0-660f356dc505\" (UID: \"c7621967-ab48-4781-bda0-660f356dc505\") "
Oct 30 20:01:26.050881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.050853    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume" pod "c7621967-ab48-4781-bda0-660f356dc505" (UID: "c7621967-ab48-4781-bda0-660f356dc505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:26.052487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.052454    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh" pod "c7621967-ab48-4781-bda0-660f356dc505" (UID: "c7621967-ab48-4781-bda0-660f356dc505"). InnerVolumeSpecName "kube-api-access-942hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:26.152117 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.152091    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7621967-ab48-4781-bda0-660f356dc505-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:26.152117 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.152116    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-942hh\" (UniqueName: \"kubernetes.io/projected/c7621967-ab48-4781-bda0-660f356dc505-kube-api-access-942hh\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:26.356142 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.356078    4320 generic.go:356] "Generic (PLEG): container finished" podID="c7621967-ab48-4781-bda0-660f356dc505" containerID="34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702" exitCode=0
Oct 30 20:01:26.356278 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.356144    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"
Oct 30 20:01:26.356278 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.356161    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" event={"ID":"c7621967-ab48-4781-bda0-660f356dc505","Type":"ContainerDied","Data":"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"}
Oct 30 20:01:26.356278 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.356223    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l" event={"ID":"c7621967-ab48-4781-bda0-660f356dc505","Type":"ContainerDied","Data":"d739c27e39eb322d82e12903b880cbf057dbea4d35fcdaaf89b32ec2a535bdd8"}
Oct 30 20:01:26.356278 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.356246    4320 scope.go:122] "RemoveContainer" containerID="34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"
Oct 30 20:01:26.365898 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.365871    4320 scope.go:122] "RemoveContainer" containerID="34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"
Oct 30 20:01:26.366166 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:26.366150    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702\": container with ID starting with 34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702 not found: ID does not exist" containerID="34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"
Oct 30 20:01:26.366212 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.366173    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702"} err="failed to get container status \"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702\": rpc error: code = NotFound desc = could not find container \"34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702\": container with ID starting with 34f9b790c143793d56efddd14c422c78e9b14d71d86c83f69425115539f16702 not found: ID does not exist"
Oct 30 20:01:26.406675 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.406642    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 20:01:26.413134 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:26.413112    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cd4bde-76679b9544-5hh8l"]
Oct 30 20:01:28.097575 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:28.097539    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7621967-ab48-4781-bda0-660f356dc505" path="/var/lib/kubelet/pods/c7621967-ab48-4781-bda0-660f356dc505/volumes"
Oct 30 20:01:28.644685 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:28.644650    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:01:31.170613 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:31.170559    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:01:39.956104 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:39.956067    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 20:01:39.956973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:39.956941    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" containerID="cri-o://a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2" gracePeriod=30
Oct 30 20:01:42.205727 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:42.205694    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 20:01:42.206380 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:42.206345    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy" containerID="cri-o://8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1" gracePeriod=30
Oct 30 20:01:43.598915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:43.598877    4320 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:43.599294 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:43.598932    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:45.240405 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.240345    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 20:01:45.372316 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372281    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372386    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372452    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372533    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372555    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372574    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372667    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372695    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372747    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.372863 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372830    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:45.373400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.372947    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\") pod \"e9353dd8-420e-469b-9c85-429d115dd184\" (UID: \"e9353dd8-420e-469b-9c85-429d115dd184\") "
Oct 30 20:01:45.373400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373098    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:45.373400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373185    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:45.373400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373348    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:45.373696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373675    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.373773 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373702    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.373773 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373719    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9353dd8-420e-469b-9c85-429d115dd184-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.373773 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373734    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.373773 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.373750    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.374915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.374890    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:45.375131 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.375102    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:45.375253 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.375201    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "kube-api-access-jhw2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:45.375312 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.375298    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token" pod "e9353dd8-420e-469b-9c85-429d115dd184" (UID: "e9353dd8-420e-469b-9c85-429d115dd184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:45.475108 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.475078    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9353dd8-420e-469b-9c85-429d115dd184-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.475251 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.475128    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhw2z\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-kube-api-access-jhw2z\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.475251 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.475144    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9353dd8-420e-469b-9c85-429d115dd184-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.475251 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.475157    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9353dd8-420e-469b-9c85-429d115dd184-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:45.541085 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.541016    4320 generic.go:356] "Generic (PLEG): container finished" podID="e9353dd8-420e-469b-9c85-429d115dd184" containerID="a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2" exitCode=0
Oct 30 20:01:45.541218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.541081    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"
Oct 30 20:01:45.541218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.541097    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerDied","Data":"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"}
Oct 30 20:01:45.541218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.541178    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk" event={"ID":"e9353dd8-420e-469b-9c85-429d115dd184","Type":"ContainerDied","Data":"132b219c8693ca774387d36c1eca493efff36363a30bf32a1e3d590dbcd254e9"}
Oct 30 20:01:45.541218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.541213    4320 scope.go:122] "RemoveContainer" containerID="a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"
Oct 30 20:01:45.551535 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.551520    4320 scope.go:122] "RemoveContainer" containerID="10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"
Oct 30 20:01:45.560556 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.560536    4320 scope.go:122] "RemoveContainer" containerID="a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"
Oct 30 20:01:45.560886 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:45.560865    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2\": container with ID starting with a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2 not found: ID does not exist" containerID="a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"
Oct 30 20:01:45.560973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.560899    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2"} err="failed to get container status \"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2\": rpc error: code = NotFound desc = could not find container \"a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2\": container with ID starting with a2bdebf60e4e525892c15680ec035a34d3fe90e588b9d8efacea9753e163e4f2 not found: ID does not exist"
Oct 30 20:01:45.560973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.560922    4320 scope.go:122] "RemoveContainer" containerID="10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"
Oct 30 20:01:45.561172 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:45.561153    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d\": container with ID starting with 10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d not found: ID does not exist" containerID="10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"
Oct 30 20:01:45.561214 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.561177    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d"} err="failed to get container status \"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d\": rpc error: code = NotFound desc = could not find container \"10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d\": container with ID starting with 10d4cbd7958717e34c59934eb4e73416e865e339bdaa6fdfe09489647bd10f6d not found: ID does not exist"
Oct 30 20:01:45.595341 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.595319    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 20:01:45.608527 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.608503    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-fkqhk"]
Oct 30 20:01:45.736995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.736964    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 20:01:45.737182 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:45.737159    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" podUID="14eb7a09-96f9-4a4e-8722-083fa758dd54" containerName="maas-api" containerID="cri-o://907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274" gracePeriod=30
Oct 30 20:01:46.063419 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.063394    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 20:01:46.098651 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.098613    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9353dd8-420e-469b-9c85-429d115dd184" path="/var/lib/kubelet/pods/e9353dd8-420e-469b-9c85-429d115dd184/volumes"
Oct 30 20:01:46.183175 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183133    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\" (UniqueName: \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\") pod \"14eb7a09-96f9-4a4e-8722-083fa758dd54\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") "
Oct 30 20:01:46.183376 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183197    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\") pod \"14eb7a09-96f9-4a4e-8722-083fa758dd54\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") "
Oct 30 20:01:46.183376 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183301    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\") pod \"14eb7a09-96f9-4a4e-8722-083fa758dd54\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") "
Oct 30 20:01:46.183530 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183425    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\") pod \"14eb7a09-96f9-4a4e-8722-083fa758dd54\" (UID: \"14eb7a09-96f9-4a4e-8722-083fa758dd54\") "
Oct 30 20:01:46.184563 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183734    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca" pod "14eb7a09-96f9-4a4e-8722-083fa758dd54" (UID: "14eb7a09-96f9-4a4e-8722-083fa758dd54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:46.184563 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.183980    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7a09-96f9-4a4e-8722-083fa758dd54-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:46.187023 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.186564    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh" pod "14eb7a09-96f9-4a4e-8722-083fa758dd54" (UID: "14eb7a09-96f9-4a4e-8722-083fa758dd54"). InnerVolumeSpecName "kube-api-access-9v4wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:46.187023 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.186955    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls" pod "14eb7a09-96f9-4a4e-8722-083fa758dd54" (UID: "14eb7a09-96f9-4a4e-8722-083fa758dd54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:46.187354 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.187308    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs" pod "14eb7a09-96f9-4a4e-8722-083fa758dd54" (UID: "14eb7a09-96f9-4a4e-8722-083fa758dd54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:46.289624 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.286551    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:46.289624 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.286590    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v4wh\" (UniqueName: \"kubernetes.io/projected/14eb7a09-96f9-4a4e-8722-083fa758dd54-kube-api-access-9v4wh\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:46.289624 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.286622    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14eb7a09-96f9-4a4e-8722-083fa758dd54-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:46.550144 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.550044    4320 generic.go:356] "Generic (PLEG): container finished" podID="14eb7a09-96f9-4a4e-8722-083fa758dd54" containerID="907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274" exitCode=0
Oct 30 20:01:46.550144 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.550122    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"
Oct 30 20:01:46.550367 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.550137    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" event={"ID":"14eb7a09-96f9-4a4e-8722-083fa758dd54","Type":"ContainerDied","Data":"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"}
Oct 30 20:01:46.550367 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.550206    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc" event={"ID":"14eb7a09-96f9-4a4e-8722-083fa758dd54","Type":"ContainerDied","Data":"1c5ae85532458c934ea21df9cd298498705bc3f37af39e52c72b22d79ab55c5c"}
Oct 30 20:01:46.550367 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.550241    4320 scope.go:122] "RemoveContainer" containerID="907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"
Oct 30 20:01:46.563799 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.563781    4320 scope.go:122] "RemoveContainer" containerID="907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"
Oct 30 20:01:46.564076 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:46.564057    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274\": container with ID starting with 907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274 not found: ID does not exist" containerID="907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"
Oct 30 20:01:46.564124 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.564086    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274"} err="failed to get container status \"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274\": rpc error: code = NotFound desc = could not find container \"907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274\": container with ID starting with 907f9e8350a187fb6b89fcbf16086797834406a48db520979d1e4d95efb67274 not found: ID does not exist"
Oct 30 20:01:46.605470 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.605442    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 20:01:46.616522 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:46.616496    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2afc11-7c4db8b886-dcpfc"]
Oct 30 20:01:47.488280 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.488215    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 20:01:47.563097 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.563056    4320 generic.go:356] "Generic (PLEG): container finished" podID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerID="8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1" exitCode=0
Oct 30 20:01:47.563295 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.563189    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" event={"ID":"a2b59786-be29-4f1c-81d5-485eae4fe8a1","Type":"ContainerDied","Data":"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"}
Oct 30 20:01:47.563295 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.563239    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg" event={"ID":"a2b59786-be29-4f1c-81d5-485eae4fe8a1","Type":"ContainerDied","Data":"33ecc072abad039632809ba4f2f434b291dda749cdb57f5d0e2aaf4210f0ed4f"}
Oct 30 20:01:47.563295 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.563267    4320 scope.go:122] "RemoveContainer" containerID="8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"
Oct 30 20:01:47.563505 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.563358    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"
Oct 30 20:01:47.586483 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.586241    4320 scope.go:122] "RemoveContainer" containerID="8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"
Oct 30 20:01:47.592149 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:47.589959    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1\": container with ID starting with 8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1 not found: ID does not exist" containerID="8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"
Oct 30 20:01:47.592149 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.589998    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1"} err="failed to get container status \"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1\": rpc error: code = NotFound desc = could not find container \"8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1\": container with ID starting with 8994544bff3250aeb5893205973bf3105606b307ffa51ab67fe58ffba22083a1 not found: ID does not exist"
Oct 30 20:01:47.603117 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603082    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603245 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603228    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603327 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603270    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603394 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603341    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603451 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603394    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603468    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603553 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603505    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.603646 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603550    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603646 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603580    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603646 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603628    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\") pod \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\" (UID: \"a2b59786-be29-4f1c-81d5-485eae4fe8a1\") "
Oct 30 20:01:47.603806 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.603747    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.604239 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.604194    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:47.604239 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.604211    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.604411 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.604238    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.604411 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.604256    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.604786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.604767    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.605918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.605882    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:47.605918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.605901    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:47.606745 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.606727    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "kube-api-access-45x88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:47.606831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.606752    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token" pod "a2b59786-be29-4f1c-81d5-485eae4fe8a1" (UID: "a2b59786-be29-4f1c-81d5-485eae4fe8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:47.705607 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705566    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45x88\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-kube-api-access-45x88\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705607 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705611    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705622    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705631    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705641    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705649    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.705788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.705658    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b59786-be29-4f1c-81d5-485eae4fe8a1-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:47.926811 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.926778    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 20:01:47.933016 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:47.932992    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-cq8wg"]
Oct 30 20:01:48.097591 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:48.097550    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14eb7a09-96f9-4a4e-8722-083fa758dd54" path="/var/lib/kubelet/pods/14eb7a09-96f9-4a4e-8722-083fa758dd54/volumes"
Oct 30 20:01:48.098139 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:48.098116    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" path="/var/lib/kubelet/pods/a2b59786-be29-4f1c-81d5-485eae4fe8a1/volumes"
Oct 30 20:01:53.997218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:53.997184    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 20:01:53.997896 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:53.997384    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy" containerID="cri-o://2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8" gracePeriod=30
Oct 30 20:01:58.645817 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:58.645734    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:01:59.263722 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.263666    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 20:01:59.313412 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313389    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313556 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313452    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313556 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313485    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313705 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313555    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313705 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313644    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313818 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313704    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313818 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313774    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313832    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.313938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313842    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.313938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.313878    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\") pod \"ff18f27f-6755-4e38-b9cc-2879a4d17529\" (UID: \"ff18f27f-6755-4e38-b9cc-2879a4d17529\") "
Oct 30 20:01:59.314114 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314051    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:59.314693 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314489    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.314941 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314708    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff18f27f-6755-4e38-b9cc-2879a4d17529-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.314941 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314898    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.314941 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314919    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.315147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.314952    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.316254 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.316030    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:59.317219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.317089    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:59.317219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.317183    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:59.318131 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.318104    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr" pod "ff18f27f-6755-4e38-b9cc-2879a4d17529" (UID: "ff18f27f-6755-4e38-b9cc-2879a4d17529"). InnerVolumeSpecName "kube-api-access-bnzfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:59.416225 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416193    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416225 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416221    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnzfr\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-kube-api-access-bnzfr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416234    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416246    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416258    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416269    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.416404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.416281    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff18f27f-6755-4e38-b9cc-2879a4d17529-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:01:59.639164 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.639132    4320 generic.go:356] "Generic (PLEG): container finished" podID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerID="2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8" exitCode=0
Oct 30 20:01:59.639300 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.639179    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"
Oct 30 20:01:59.639300 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.639215    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerDied","Data":"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"}
Oct 30 20:01:59.639300 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.639277    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl" event={"ID":"ff18f27f-6755-4e38-b9cc-2879a4d17529","Type":"ContainerDied","Data":"bc24c66d4fba1870594cdcf1eee88c9eb2ce9948e565c5268cd64bcf77c3d6ce"}
Oct 30 20:01:59.639417 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.639308    4320 scope.go:122] "RemoveContainer" containerID="2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"
Oct 30 20:01:59.650541 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.650524    4320 scope.go:122] "RemoveContainer" containerID="8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"
Oct 30 20:01:59.659094 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.659072    4320 scope.go:122] "RemoveContainer" containerID="2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"
Oct 30 20:01:59.659370 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:59.659351    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8\": container with ID starting with 2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8 not found: ID does not exist" containerID="2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"
Oct 30 20:01:59.659440 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.659380    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8"} err="failed to get container status \"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8\": rpc error: code = NotFound desc = could not find container \"2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8\": container with ID starting with 2cb624dbd018145c21a736a48edba092f8d820139a23d33ced7d862b678ca3c8 not found: ID does not exist"
Oct 30 20:01:59.659440 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.659396    4320 scope.go:122] "RemoveContainer" containerID="8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"
Oct 30 20:01:59.659672 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:01:59.659652    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec\": container with ID starting with 8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec not found: ID does not exist" containerID="8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"
Oct 30 20:01:59.659723 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.659680    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec"} err="failed to get container status \"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec\": rpc error: code = NotFound desc = could not find container \"8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec\": container with ID starting with 8aa04d3c3d240b3ac500ff7b9623c339866d6f5317d603a97a4e982523bc32ec not found: ID does not exist"
Oct 30 20:01:59.692143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.692118    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 20:01:59.722800 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.721568    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2afc11-openshift-default-5f55d5dcb9-4chnl"]
Oct 30 20:01:59.761156 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.761126    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 20:01:59.761421 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:01:59.761374    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" containerID="cri-o://654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13" gracePeriod=30
Oct 30 20:02:00.096051 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:00.095971    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" path="/var/lib/kubelet/pods/ff18f27f-6755-4e38-b9cc-2879a4d17529/volumes"
Oct 30 20:02:04.096182 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:04.096154    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:02:04.929376 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:04.929336    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:02:04.929579 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:04.929414    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 20:02:05.056648 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.056622    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 20:02:05.194563 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194531    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194575    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194648    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194694    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194727    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194771    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194800    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194857    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194921    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194961    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:05.195075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.194985    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\") pod \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\" (UID: \"b7a86e68-8ece-4fee-834e-b8a8c492cbcc\") "
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195182    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195281    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195505    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195563    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195585    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195636    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.195676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.195652    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.197103 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.197076    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:05.197340 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.197317    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:05.197573 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.197552    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "kube-api-access-m9msc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:05.197671 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.197582    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token" pod "b7a86e68-8ece-4fee-834e-b8a8c492cbcc" (UID: "b7a86e68-8ece-4fee-834e-b8a8c492cbcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:05.304775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.300557    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.304775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.300616    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.304775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.300634    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.304775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.300648    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.304775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.300674    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9msc\" (UniqueName: \"kubernetes.io/projected/b7a86e68-8ece-4fee-834e-b8a8c492cbcc-kube-api-access-m9msc\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:05.717300 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.717234    4320 generic.go:356] "Generic (PLEG): container finished" podID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerID="654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13" exitCode=0
Oct 30 20:02:05.717504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.717380    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" event={"ID":"b7a86e68-8ece-4fee-834e-b8a8c492cbcc","Type":"ContainerDied","Data":"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"}
Oct 30 20:02:05.717504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.717439    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc" event={"ID":"b7a86e68-8ece-4fee-834e-b8a8c492cbcc","Type":"ContainerDied","Data":"615c5a25a623e173c6840439d5cf33ee91fb96131dce1178b6cd9be096c14690"}
Oct 30 20:02:05.717504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.717469    4320 scope.go:122] "RemoveContainer" containerID="654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"
Oct 30 20:02:05.717726 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.717656    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"
Oct 30 20:02:05.730956 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.730938    4320 scope.go:122] "RemoveContainer" containerID="654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"
Oct 30 20:02:05.731247 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:05.731227    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13\": container with ID starting with 654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13 not found: ID does not exist" containerID="654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"
Oct 30 20:02:05.731296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.731254    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13"} err="failed to get container status \"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13\": rpc error: code = NotFound desc = could not find container \"654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13\": container with ID starting with 654ac22e6a6ea12657f6e76917943d0f8a4d62223aaa8722540167bd6a1e8b13 not found: ID does not exist"
Oct 30 20:02:05.784167 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.784116    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 20:02:05.796013 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.795989    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-cjnfc"]
Oct 30 20:02:05.852550 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.852527    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 20:02:05.852777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:05.852738    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy" containerID="cri-o://1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237" gracePeriod=30
Oct 30 20:02:06.096129 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:06.096055    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" path="/var/lib/kubelet/pods/b7a86e68-8ece-4fee-834e-b8a8c492cbcc/volumes"
Oct 30 20:02:10.070700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:10.070614    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 20:02:10.071241 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:10.070793    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" containerID="cri-o://4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05" gracePeriod=30
Oct 30 20:02:11.119576 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.119522    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 20:02:11.274007 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.273973    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274209 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274064    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274209 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274115    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274209 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274186    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274275    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274297    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.274414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274314    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274414 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274354    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274433    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.274669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.274465    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\") pod \"132e6434-8b27-4a54-a4dd-f181578771cf\" (UID: \"132e6434-8b27-4a54-a4dd-f181578771cf\") "
Oct 30 20:02:11.275972 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.275260    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.275972 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.275354    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.275972 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.275691    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:11.275972 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.275735    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.276879 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.276847    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "kube-api-access-4bwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:11.282629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.277493    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.282629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.277521    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.282629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.277548    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.282629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.277563    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e6434-8b27-4a54-a4dd-f181578771cf-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.282629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.277578    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.289573 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.289539    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.290356 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.290334    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:11.290919 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.290892    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token" pod "132e6434-8b27-4a54-a4dd-f181578771cf" (UID: "132e6434-8b27-4a54-a4dd-f181578771cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:11.386724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.382643    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.386724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.382672    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwmg\" (UniqueName: \"kubernetes.io/projected/132e6434-8b27-4a54-a4dd-f181578771cf-kube-api-access-4bwmg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.386724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.382683    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e6434-8b27-4a54-a4dd-f181578771cf-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.386724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.382692    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e6434-8b27-4a54-a4dd-f181578771cf-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:11.755003 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.754919    4320 generic.go:356] "Generic (PLEG): container finished" podID="132e6434-8b27-4a54-a4dd-f181578771cf" containerID="1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237" exitCode=0
Oct 30 20:02:11.755003 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.754969    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" event={"ID":"132e6434-8b27-4a54-a4dd-f181578771cf","Type":"ContainerDied","Data":"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"}
Oct 30 20:02:11.755204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.755010    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"
Oct 30 20:02:11.755204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.755076    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j" event={"ID":"132e6434-8b27-4a54-a4dd-f181578771cf","Type":"ContainerDied","Data":"d4691c89e655d4526dd65a3eef4a7f41b8adfa8d023fae2943be62b1947f8664"}
Oct 30 20:02:11.755204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.755111    4320 scope.go:122] "RemoveContainer" containerID="1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"
Oct 30 20:02:11.766763 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.766738    4320 scope.go:122] "RemoveContainer" containerID="1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"
Oct 30 20:02:11.767084 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:11.767057    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237\": container with ID starting with 1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237 not found: ID does not exist" containerID="1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"
Oct 30 20:02:11.767157 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.767098    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237"} err="failed to get container status \"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237\": rpc error: code = NotFound desc = could not find container \"1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237\": container with ID starting with 1d4ec1fe2185d6a1347d78b30d439a2cb33265aff2e0024b167b2f8df421f237 not found: ID does not exist"
Oct 30 20:02:11.806695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.806664    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 20:02:11.818118 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:11.818085    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-8zl5j"]
Oct 30 20:02:12.110040 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:12.109962    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" path="/var/lib/kubelet/pods/132e6434-8b27-4a54-a4dd-f181578771cf/volumes"
Oct 30 20:02:13.254836 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:13.254798    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8k46t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:13.255217 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:13.254856    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:15.431172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.431149    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 20:02:15.451671 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451644    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451812 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451685    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451812 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451723    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451812 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451767    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451831    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451857    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451885    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451944    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.451995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.451981    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\") pod \"7a12a296-1061-4e61-8ff8-c6a63a5320da\" (UID: \"7a12a296-1061-4e61-8ff8-c6a63a5320da\") "
Oct 30 20:02:15.452246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452033    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:15.452246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452123    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:15.452246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452167    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:15.452246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452230    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:15.452406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452320    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:15.452701 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452680    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.452843 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452823    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a12a296-1061-4e61-8ff8-c6a63a5320da-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.452916 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452854    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.452916 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452872    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.452916 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.452886    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.454010 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.453985    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:15.454275 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.454256    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:15.454366 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.454345    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "kube-api-access-rb49w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:15.454497 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.454483    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo" pod "7a12a296-1061-4e61-8ff8-c6a63a5320da" (UID: "7a12a296-1061-4e61-8ff8-c6a63a5320da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:15.554184 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.554152    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.554184 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.554179    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.554184 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.554188    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb49w\" (UniqueName: \"kubernetes.io/projected/7a12a296-1061-4e61-8ff8-c6a63a5320da-kube-api-access-rb49w\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.554400 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.554197    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a12a296-1061-4e61-8ff8-c6a63a5320da-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:15.783691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.783609    4320 generic.go:356] "Generic (PLEG): container finished" podID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerID="4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05" exitCode=0
Oct 30 20:02:15.783691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.783626    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" event={"ID":"7a12a296-1061-4e61-8ff8-c6a63a5320da","Type":"ContainerDied","Data":"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"}
Oct 30 20:02:15.783691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.783661    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"
Oct 30 20:02:15.783691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.783692    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t" event={"ID":"7a12a296-1061-4e61-8ff8-c6a63a5320da","Type":"ContainerDied","Data":"8b546c82e82a053bf21aeb024d8b7a41352301101e5d6b21911d02f75a8f094b"}
Oct 30 20:02:15.783991 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.783715    4320 scope.go:122] "RemoveContainer" containerID="4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"
Oct 30 20:02:15.794440 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.794425    4320 scope.go:122] "RemoveContainer" containerID="4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"
Oct 30 20:02:15.794737 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:15.794719    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05\": container with ID starting with 4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05 not found: ID does not exist" containerID="4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"
Oct 30 20:02:15.794806 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.794744    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05"} err="failed to get container status \"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05\": rpc error: code = NotFound desc = could not find container \"4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05\": container with ID starting with 4a28e84cfb0acb331d6aba9852717046db575bdd3bb65609d50c57b8f2bacd05 not found: ID does not exist"
Oct 30 20:02:15.834855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.834830    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 20:02:15.838225 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:15.838204    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8k46t"]
Oct 30 20:02:16.097439 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:16.097367    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" path="/var/lib/kubelet/pods/7a12a296-1061-4e61-8ff8-c6a63a5320da/volumes"
Oct 30 20:02:23.028778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:23.028746    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"]
Oct 30 20:02:23.044150 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:23.044112    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5418cce-54v58"]
Oct 30 20:02:24.027193 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:24.027163    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"]
Oct 30 20:02:24.038529 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:24.038499    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6fd5ef-b42dw"]
Oct 30 20:02:24.096065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:24.096037    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3fd2164-2685-43de-b9f6-e304122c6dda" path="/var/lib/kubelet/pods/b3fd2164-2685-43de-b9f6-e304122c6dda/volumes"
Oct 30 20:02:24.096497 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:24.096481    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fa1235-87ce-4ed4-b754-070eaf825881" path="/var/lib/kubelet/pods/b9fa1235-87ce-4ed4-b754-070eaf825881/volumes"
Oct 30 20:02:26.032510 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.032478    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 20:02:26.033187 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.032697    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerName="maas-api" containerID="cri-o://fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4" gracePeriod=30
Oct 30 20:02:26.339716 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.339686    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 20:02:26.376511 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.376477    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\") pod \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") "
Oct 30 20:02:26.376700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.376591    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\") pod \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") "
Oct 30 20:02:26.376771 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.376701    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\" (UniqueName: \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\") pod \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") "
Oct 30 20:02:26.376837 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.376817    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\") pod \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\" (UID: \"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687\") "
Oct 30 20:02:26.376955 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.376924    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca" pod "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" (UID: "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:26.377296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.377270    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:26.380673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.380014    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls" pod "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" (UID: "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:26.380673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.380142    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv" pod "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" (UID: "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687"). InnerVolumeSpecName "kube-api-access-jgzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:26.386341 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.386316    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs" pod "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" (UID: "f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:26.478211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.478184    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:26.478211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.478206    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:26.478360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.478218    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzlv\" (UniqueName: \"kubernetes.io/projected/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687-kube-api-access-jgzlv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:26.878844 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.878807    4320 generic.go:356] "Generic (PLEG): container finished" podID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerID="fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4" exitCode=0
Oct 30 20:02:26.879032 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.878854    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"
Oct 30 20:02:26.879032 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.878881    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" event={"ID":"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687","Type":"ContainerDied","Data":"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"}
Oct 30 20:02:26.879032 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.878947    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" event={"ID":"f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687","Type":"ContainerDied","Data":"538a0ea7d27547eccb32df650582622d648a063292a99abd3bb33302c46c889b"}
Oct 30 20:02:26.879032 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.878971    4320 scope.go:122] "RemoveContainer" containerID="fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"
Oct 30 20:02:26.889915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.889899    4320 scope.go:122] "RemoveContainer" containerID="fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"
Oct 30 20:02:26.890188 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:26.890169    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4\": container with ID starting with fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4 not found: ID does not exist" containerID="fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"
Oct 30 20:02:26.890263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.890195    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4"} err="failed to get container status \"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4\": rpc error: code = NotFound desc = could not find container \"fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4\": container with ID starting with fd46281135ad9e16fa1949a46ef8e121060d4399668a654681a6e2ddee908de4 not found: ID does not exist"
Oct 30 20:02:26.935102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.935065    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 20:02:26.937886 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:26.937861    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m"]
Oct 30 20:02:28.096061 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:28.096028    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" path="/var/lib/kubelet/pods/f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687/volumes"
Oct 30 20:02:28.581212 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:28.581173    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 20:02:28.581436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:28.581409    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" containerID="cri-o://7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf" gracePeriod=30
Oct 30 20:02:28.649662 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:28.649625    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:02:29.336741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:29.336668    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fe3e5e-777f55c84b-cx74m" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.203:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:33.266875 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.266841    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:33.267282 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.266895    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:33.848143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.848089    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 20:02:33.941038 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.941007    4320 generic.go:356] "Generic (PLEG): container finished" podID="8f113b99-eb42-4364-b901-37eead38c18f" containerID="7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf" exitCode=0
Oct 30 20:02:33.941215 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.941055    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"
Oct 30 20:02:33.941215 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.941077    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" event={"ID":"8f113b99-eb42-4364-b901-37eead38c18f","Type":"ContainerDied","Data":"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"}
Oct 30 20:02:33.941215 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.941121    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6" event={"ID":"8f113b99-eb42-4364-b901-37eead38c18f","Type":"ContainerDied","Data":"385c4cfa6112578911dd00bb1f361c1c17969ad4266086172c65241c7cffe620"}
Oct 30 20:02:33.941215 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.941145    4320 scope.go:122] "RemoveContainer" containerID="7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"
Oct 30 20:02:33.950472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.950451    4320 scope.go:122] "RemoveContainer" containerID="7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"
Oct 30 20:02:33.950802 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:33.950778    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf\": container with ID starting with 7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf not found: ID does not exist" containerID="7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"
Oct 30 20:02:33.950910 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.950809    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf"} err="failed to get container status \"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf\": rpc error: code = NotFound desc = could not find container \"7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf\": container with ID starting with 7b1d0f48344c9a7705513bd5d28ebd0309abc2d52683df46feb4fa095716a4cf not found: ID does not exist"
Oct 30 20:02:33.956006 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.955985    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956023    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956070    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956178 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956132    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956259 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956243    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956308 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956281    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956372 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956354    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956388    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956371    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.956552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956449    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.956552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956506    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\") pod \"8f113b99-eb42-4364-b901-37eead38c18f\" (UID: \"8f113b99-eb42-4364-b901-37eead38c18f\") "
Oct 30 20:02:33.956694 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956658    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.956918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956874    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:33.956918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.956892    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:33.957187 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.957169    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:33.957224 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.957196    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f113b99-eb42-4364-b901-37eead38c18f-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:33.957224 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.957209    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:33.957301 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.957222    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:33.957301 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.957240    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:33.958372 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.958346    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:33.958455 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.958433    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:33.958681 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.958663    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "kube-api-access-9t8pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:33.958782 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:33.958764    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy" pod "8f113b99-eb42-4364-b901-37eead38c18f" (UID: "8f113b99-eb42-4364-b901-37eead38c18f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:34.058436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.058406    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8pr\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-kube-api-access-9t8pr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.058436 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.058433    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f113b99-eb42-4364-b901-37eead38c18f-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.058626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.058448    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f113b99-eb42-4364-b901-37eead38c18f-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.058626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.058459    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f113b99-eb42-4364-b901-37eead38c18f-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.075084 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.075056    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 20:02:34.279658 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.279608    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 20:02:34.282956 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:34.282932    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-9mwx6"]
Oct 30 20:02:36.097246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:36.097211    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f113b99-eb42-4364-b901-37eead38c18f" path="/var/lib/kubelet/pods/8f113b99-eb42-4364-b901-37eead38c18f/volumes"
Oct 30 20:02:36.268086 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:36.268041    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 20:02:36.339117 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:36.339088    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:02:39.825781 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:39.825746    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 20:02:41.268961 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:41.268895    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing" containerID="cri-o://8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6" gracePeriod=25
Oct 30 20:02:44.595489 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.595451    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"]
Oct 30 20:02:44.596231 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596209    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:02:44.596276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596235    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:02:44.596276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596253    4320 cpu_manager.go:414] "containerMap: removing container" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.596276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596260    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596275    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596283    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596299    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596306    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596317    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596324    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596337    4320 cpu_manager.go:414] "containerMap: removing container" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596344    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596353    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596362    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596374    4320 cpu_manager.go:414] "containerMap: removing container" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596381    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596392    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596400    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596413    4320 cpu_manager.go:414] "containerMap: removing container" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:02:44.596426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596420    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596435    4320 cpu_manager.go:414] "containerMap: removing container" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596441    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596453    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596460    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596476    4320 cpu_manager.go:414] "containerMap: removing container" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596483    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596503    4320 cpu_manager.go:414] "containerMap: removing container" podUID="14eb7a09-96f9-4a4e-8722-083fa758dd54" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596510    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14eb7a09-96f9-4a4e-8722-083fa758dd54" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596521    4320 cpu_manager.go:414] "containerMap: removing container" podUID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596528    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596542    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596550    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596563    4320 cpu_manager.go:414] "containerMap: removing container" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596569    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596584    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596591    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596617    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596624    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596635    4320 cpu_manager.go:414] "containerMap: removing container" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596643    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596654    4320 cpu_manager.go:414] "containerMap: removing container" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596661    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596672    4320 cpu_manager.go:414] "containerMap: removing container" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596680    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596695    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596703    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596716    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596723    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596740    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c7621967-ab48-4781-bda0-660f356dc505" containerName="payload-processing"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596748    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7621967-ab48-4781-bda0-660f356dc505" containerName="payload-processing"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596759    4320 cpu_manager.go:414] "containerMap: removing container" podUID="198b2307-c817-4ed3-b88e-7d07b3383357" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596766    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198b2307-c817-4ed3-b88e-7d07b3383357" containerName="maas-api"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596777    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596784    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596797    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596815    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596829    4320 cpu_manager.go:414] "containerMap: removing container" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596835    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596846    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596854    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596865    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596871    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596887    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596893    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596907    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596914    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596928    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596935    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596943    4320 cpu_manager.go:414] "containerMap: removing container" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596950    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596960    4320 cpu_manager.go:414] "containerMap: removing container" podUID="59e8727b-6ad8-4555-9257-3235e7e9375a" containerName="revoke-keys"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596967    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e8727b-6ad8-4555-9257-3235e7e9375a" containerName="revoke-keys"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596977    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.596985    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597122    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9cfbc6-f03b-4a46-adf5-9da7ab25a150" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597135    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597144    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eb04e2-4ee0-4fc0-bdc8-9db4110552ba" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597153    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597161    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597170    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="82404b8d-352d-487b-991f-5931d7c74b1a" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597179    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597187    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597195    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fab2ed-de59-4829-88f8-547e0c87e610" containerName="main"
Oct 30 20:02:44.597169 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597205    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="99864e99-111a-44e4-bd64-8be3c2ec3387" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597216    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="132e6434-8b27-4a54-a4dd-f181578771cf" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597228    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c6d48c-dfc7-4aae-adbd-fd49ab1ee687" containerName="maas-api"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597236    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2371dc5c-ddc4-44f9-a83f-d2753aa58873" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597244    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597254    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7621967-ab48-4781-bda0-660f356dc505" containerName="payload-processing"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597263    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="198b2307-c817-4ed3-b88e-7d07b3383357" containerName="maas-api"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597271    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597279    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597286    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="314e2376-43ec-4006-a31e-5d426ebbf9a2" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597295    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a12a296-1061-4e61-8ff8-c6a63a5320da" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597303    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597311    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b59786-be29-4f1c-81d5-485eae4fe8a1" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597322    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f113b99-eb42-4364-b901-37eead38c18f" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597331    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597340    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597349    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597356    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e8727b-6ad8-4555-9257-3235e7e9375a" containerName="revoke-keys"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597365    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597375    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597383    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597390    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="09713a4d-4542-4978-a9e8-9f2915794d50" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597398    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55e4284-8465-45aa-b169-056a83859ed4" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597406    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597412    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597424    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac09df1-8a18-487c-a762-7d47949ff198" containerName="main"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597433    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec81397-7f07-43b8-a73f-acae90decc27" containerName="cleanup"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597440    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a86e68-8ece-4fee-834e-b8a8c492cbcc" containerName="istio-proxy"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597452    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="14eb7a09-96f9-4a4e-8722-083fa758dd54" containerName="maas-api"
Oct 30 20:02:44.599673 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.597462    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="30587b0d-6cad-4690-8daa-03b6f2ccbfc5" containerName="main"
Oct 30 20:02:44.602707 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.602676    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.619971 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.619010    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"]
Oct 30 20:02:44.625821 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.625797    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:44.625967 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.625855    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:02:44.695388 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.695357    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.695550 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.695424    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.695550 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.695523    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62kz\" (UniqueName: \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.796249 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.796214    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62kz\" (UniqueName: \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.796405 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.796327    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.796405 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.796394    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.797089 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.797062    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.798678 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.798647    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.816113 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.816086    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62kz\" (UniqueName: \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\") pod \"maas-api-revoke-sub-22c73a9d-hdt97\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:44.826422 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.826379    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing" containerID="cri-o://3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003" gracePeriod=25
Oct 30 20:02:44.921195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:44.921171    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:45.254927 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:45.254392    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"]
Oct 30 20:02:45.257922 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:02:45.257886    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80984c1d_f330_4004_8086_d143a6a6e93e.slice/crio-986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e WatchSource:0}: Error finding container 986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e: Status 404 returned error can't find the container with id 986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e
Oct 30 20:02:46.033968 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:46.033332    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97" event={"ID":"80984c1d-f330-4004-8086-d143a6a6e93e","Type":"ContainerStarted","Data":"907949f6d553b64e80a891776307f369ee7e1e4307cb8e16322f5c3e410d4bf5"}
Oct 30 20:02:46.033968 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:46.033400    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97" event={"ID":"80984c1d-f330-4004-8086-d143a6a6e93e","Type":"ContainerStarted","Data":"986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e"}
Oct 30 20:02:46.073091 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:46.073031    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97" podStartSLOduration=2.073014879 podStartE2EDuration="2.073014879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:46.072153841 +0000 UTC m=+3514.107666306" watchObservedRunningTime="2026-10-30 20:02:46.073014879 +0000 UTC m=+3514.108527343"
Oct 30 20:02:47.060831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:47.060793    4320 generic.go:356] "Generic (PLEG): container finished" podID="80984c1d-f330-4004-8086-d143a6a6e93e" containerID="907949f6d553b64e80a891776307f369ee7e1e4307cb8e16322f5c3e410d4bf5" exitCode=0
Oct 30 20:02:47.061316 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:47.060984    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97" event={"ID":"80984c1d-f330-4004-8086-d143a6a6e93e","Type":"ContainerDied","Data":"907949f6d553b64e80a891776307f369ee7e1e4307cb8e16322f5c3e410d4bf5"}
Oct 30 20:02:47.794761 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:47.794723    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:02:48.218392 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.218368    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:48.332754 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.332688    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\") pod \"80984c1d-f330-4004-8086-d143a6a6e93e\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") "
Oct 30 20:02:48.332891 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.332771    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\") pod \"80984c1d-f330-4004-8086-d143a6a6e93e\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") "
Oct 30 20:02:48.332891 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.332806    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\" (UniqueName: \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\") pod \"80984c1d-f330-4004-8086-d143a6a6e93e\" (UID: \"80984c1d-f330-4004-8086-d143a6a6e93e\") "
Oct 30 20:02:48.333105 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.333078    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca" pod "80984c1d-f330-4004-8086-d143a6a6e93e" (UID: "80984c1d-f330-4004-8086-d143a6a6e93e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:48.334807 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.334781    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca" pod "80984c1d-f330-4004-8086-d143a6a6e93e" (UID: "80984c1d-f330-4004-8086-d143a6a6e93e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:48.334924 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.334855    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz" pod "80984c1d-f330-4004-8086-d143a6a6e93e" (UID: "80984c1d-f330-4004-8086-d143a6a6e93e"). InnerVolumeSpecName "kube-api-access-d62kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:48.433745 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.433719    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:48.433745 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.433742    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80984c1d-f330-4004-8086-d143a6a6e93e-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:48.433905 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:48.433751    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62kz\" (UniqueName: \"kubernetes.io/projected/80984c1d-f330-4004-8086-d143a6a6e93e-kube-api-access-d62kz\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.074210 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.074174    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"
Oct 30 20:02:49.074210 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.074177    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97" event={"ID":"80984c1d-f330-4004-8086-d143a6a6e93e","Type":"ContainerDied","Data":"986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e"}
Oct 30 20:02:49.074474 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.074239    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986c9e4e6084f7e71be93088508b034c38c9f6514784c5dc0c6f3a80db4f5e7e"
Oct 30 20:02:49.097164 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.097115    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main" containerID="cri-o://6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3" gracePeriod=45
Oct 30 20:02:49.345467 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.345441    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 20:02:49.453119 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453083    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.453363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453175    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.453363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453330    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.453584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453369    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\" (UniqueName: \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.453584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453410    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.453584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.453516    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\") pod \"6c11a744-2060-412d-9f50-62a2b28bb993\" (UID: \"6c11a744-2060-412d-9f50-62a2b28bb993\") "
Oct 30 20:02:49.454257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.454235    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:49.454479 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.454453    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:49.454775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.454740    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:49.456959 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.456932    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:49.457064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.457012    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:49.457629 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.457580    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4" pod "6c11a744-2060-412d-9f50-62a2b28bb993" (UID: "6c11a744-2060-412d-9f50-62a2b28bb993"). InnerVolumeSpecName "kube-api-access-88mj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:49.554691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554663    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.554691 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554692    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.554880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554703    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.554880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554711    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mj4\" (UniqueName: \"kubernetes.io/projected/6c11a744-2060-412d-9f50-62a2b28bb993-kube-api-access-88mj4\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.554880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554719    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c11a744-2060-412d-9f50-62a2b28bb993-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.554880 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.554729    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c11a744-2060-412d-9f50-62a2b28bb993-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:49.701040 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:49.701004    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:02:50.081168 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.081084    4320 generic.go:356] "Generic (PLEG): container finished" podID="6c11a744-2060-412d-9f50-62a2b28bb993" containerID="6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3" exitCode=0
Oct 30 20:02:50.081168 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.081130    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"
Oct 30 20:02:50.081398 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.081176    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" event={"ID":"6c11a744-2060-412d-9f50-62a2b28bb993","Type":"ContainerDied","Data":"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"}
Oct 30 20:02:50.081398 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.081230    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg" event={"ID":"6c11a744-2060-412d-9f50-62a2b28bb993","Type":"ContainerDied","Data":"073d941339add4f67105a323f494c0853ea7e36cde0db290d8a73a4bd7b49a51"}
Oct 30 20:02:50.081398 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.081263    4320 scope.go:122] "RemoveContainer" containerID="6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"
Oct 30 20:02:50.091410 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.091387    4320 scope.go:122] "RemoveContainer" containerID="6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"
Oct 30 20:02:50.091692 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:50.091672    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3\": container with ID starting with 6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3 not found: ID does not exist" containerID="6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"
Oct 30 20:02:50.091775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.091699    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3"} err="failed to get container status \"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3\": rpc error: code = NotFound desc = could not find container \"6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3\": container with ID starting with 6b195f5cf5ccc79013a2dfd09ea17c52e83672071900c69cb9a4f64a165e3cb3 not found: ID does not exist"
Oct 30 20:02:50.131504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.131482    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 20:02:50.148622 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:50.146227    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5449c/premium-w0-c5449c-kserve-777559f966-zcjwg"]
Oct 30 20:02:52.096202 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:52.096168    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" path="/var/lib/kubelet/pods/6c11a744-2060-412d-9f50-62a2b28bb993/volumes"
Oct 30 20:02:53.779049 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:53.779015    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 20:02:53.779571 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:53.779258    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" podUID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" containerName="maas-api" containerID="cri-o://5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965" gracePeriod=30
Oct 30 20:02:54.104219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.104190    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 20:02:54.106258 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.106230    4320 generic.go:356] "Generic (PLEG): container finished" podID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" containerID="5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965" exitCode=0
Oct 30 20:02:54.106382 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.106266    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"
Oct 30 20:02:54.106382 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.106306    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" event={"ID":"414ca6a0-ed48-4c50-851d-8e2063ceed6a","Type":"ContainerDied","Data":"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"}
Oct 30 20:02:54.106382 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.106364    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds" event={"ID":"414ca6a0-ed48-4c50-851d-8e2063ceed6a","Type":"ContainerDied","Data":"ecddfd8d26b21bd27d8505e2cd83f5d6303459731134dc545cb892bdbe7a6e0a"}
Oct 30 20:02:54.106518 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.106390    4320 scope.go:122] "RemoveContainer" containerID="5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"
Oct 30 20:02:54.117777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.117754    4320 scope.go:122] "RemoveContainer" containerID="5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"
Oct 30 20:02:54.118240 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:54.118216    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965\": container with ID starting with 5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965 not found: ID does not exist" containerID="5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"
Oct 30 20:02:54.118337 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.118252    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965"} err="failed to get container status \"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965\": rpc error: code = NotFound desc = could not find container \"5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965\": container with ID starting with 5af8440405568d0717cdda6f9ea2f04946707f69a4ae044aec4f6ed2b2a02965 not found: ID does not exist"
Oct 30 20:02:54.214498 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.214467    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\" (UniqueName: \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\") pod \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") "
Oct 30 20:02:54.214699 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.214575    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\") pod \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") "
Oct 30 20:02:54.214699 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.214629    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\") pod \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") "
Oct 30 20:02:54.214699 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.214668    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\") pod \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\" (UID: \"414ca6a0-ed48-4c50-851d-8e2063ceed6a\") "
Oct 30 20:02:54.216374 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.216339    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca" pod "414ca6a0-ed48-4c50-851d-8e2063ceed6a" (UID: "414ca6a0-ed48-4c50-851d-8e2063ceed6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:54.217908 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.217876    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls" pod "414ca6a0-ed48-4c50-851d-8e2063ceed6a" (UID: "414ca6a0-ed48-4c50-851d-8e2063ceed6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:54.218402 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.218374    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2" pod "414ca6a0-ed48-4c50-851d-8e2063ceed6a" (UID: "414ca6a0-ed48-4c50-851d-8e2063ceed6a"). InnerVolumeSpecName "kube-api-access-d9mw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:54.218480 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.218398    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs" pod "414ca6a0-ed48-4c50-851d-8e2063ceed6a" (UID: "414ca6a0-ed48-4c50-851d-8e2063ceed6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:54.316001 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.315969    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mw2\" (UniqueName: \"kubernetes.io/projected/414ca6a0-ed48-4c50-851d-8e2063ceed6a-kube-api-access-d9mw2\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:54.316001 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.315999    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:54.316001 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.316009    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ca6a0-ed48-4c50-851d-8e2063ceed6a-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:54.316228 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.316019    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ca6a0-ed48-4c50-851d-8e2063ceed6a-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:02:54.453025 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.452997    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 20:02:54.458395 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:54.458366    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d9eb76-7bffb9f79f-mgzds"]
Oct 30 20:02:56.095339 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:56.095307    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" path="/var/lib/kubelet/pods/414ca6a0-ed48-4c50-851d-8e2063ceed6a/volumes"
Oct 30 20:02:56.157565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:56.157531    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:02:57.795219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:57.795179    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:02:58.644006 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:02:58.643963    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:02:59.704177 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:02:59.704123    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:03:00.915230 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:00.915192    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 20:03:00.915898 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:00.915868    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" podUID="7325a073-a722-4a52-92de-24cb960845db" containerName="maas-api" containerID="cri-o://837790640aa43186118e290c921d7b76ce0b022f97ecf640f3aff7ef699d1b20" gracePeriod=30
Oct 30 20:03:01.220741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.220480    4320 generic.go:356] "Generic (PLEG): container finished" podID="7325a073-a722-4a52-92de-24cb960845db" containerID="837790640aa43186118e290c921d7b76ce0b022f97ecf640f3aff7ef699d1b20" exitCode=0
Oct 30 20:03:01.220741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.220608    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" event={"ID":"7325a073-a722-4a52-92de-24cb960845db","Type":"ContainerDied","Data":"837790640aa43186118e290c921d7b76ce0b022f97ecf640f3aff7ef699d1b20"}
Oct 30 20:03:01.263676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.263651    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 20:03:01.420638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.418868    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\" (UniqueName: \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\") pod \"7325a073-a722-4a52-92de-24cb960845db\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") "
Oct 30 20:03:01.420638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.418965    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\") pod \"7325a073-a722-4a52-92de-24cb960845db\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") "
Oct 30 20:03:01.420638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.419012    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") pod \"7325a073-a722-4a52-92de-24cb960845db\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") "
Oct 30 20:03:01.420638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.419145    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\") pod \"7325a073-a722-4a52-92de-24cb960845db\" (UID: \"7325a073-a722-4a52-92de-24cb960845db\") "
Oct 30 20:03:01.420638 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.420188    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca" pod "7325a073-a722-4a52-92de-24cb960845db" (UID: "7325a073-a722-4a52-92de-24cb960845db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:01.422041 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.422016    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs" pod "7325a073-a722-4a52-92de-24cb960845db" (UID: "7325a073-a722-4a52-92de-24cb960845db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:01.423064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.423028    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d" pod "7325a073-a722-4a52-92de-24cb960845db" (UID: "7325a073-a722-4a52-92de-24cb960845db"). InnerVolumeSpecName "kube-api-access-qjg8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:01.426908 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.426887    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls" pod "7325a073-a722-4a52-92de-24cb960845db" (UID: "7325a073-a722-4a52-92de-24cb960845db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:01.520362 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.520333    4320 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7325a073-a722-4a52-92de-24cb960845db-openshift-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:01.520362 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.520363    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjg8d\" (UniqueName: \"kubernetes.io/projected/7325a073-a722-4a52-92de-24cb960845db-kube-api-access-qjg8d\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:01.520557 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.520376    4320 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-metrics-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:01.520557 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.520388    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7325a073-a722-4a52-92de-24cb960845db-maas-api-tls\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:01.524129 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.524109    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 20:03:01.620772 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.620744    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\" (UniqueName: \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\") pod \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") "
Oct 30 20:03:01.620963 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.620867    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\") pod \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\" (UID: \"b06ceee5-3a98-4928-ac9f-e7e9cadeebca\") "
Oct 30 20:03:01.621592 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.621564    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume" pod "b06ceee5-3a98-4928-ac9f-e7e9cadeebca" (UID: "b06ceee5-3a98-4928-ac9f-e7e9cadeebca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:01.623648 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.623622    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf" pod "b06ceee5-3a98-4928-ac9f-e7e9cadeebca" (UID: "b06ceee5-3a98-4928-ac9f-e7e9cadeebca"). InnerVolumeSpecName "kube-api-access-zbkpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:01.695472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.695436    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 20:03:01.695711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.695683    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" containerID="cri-o://ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6" gracePeriod=30
Oct 30 20:03:01.722350 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.722324    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:01.722350 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:01.722350    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkpf\" (UniqueName: \"kubernetes.io/projected/b06ceee5-3a98-4928-ac9f-e7e9cadeebca-kube-api-access-zbkpf\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:02.226984 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.226915    4320 generic.go:356] "Generic (PLEG): container finished" podID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerID="8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6" exitCode=0
Oct 30 20:03:02.226984 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.226970    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"
Oct 30 20:03:02.227416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.227000    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" event={"ID":"b06ceee5-3a98-4928-ac9f-e7e9cadeebca","Type":"ContainerDied","Data":"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"}
Oct 30 20:03:02.227416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.227063    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb" event={"ID":"b06ceee5-3a98-4928-ac9f-e7e9cadeebca","Type":"ContainerDied","Data":"c2873305ec2b3ae89763d58f96b4b64820b2d67759d9915a1eac518118eddfd0"}
Oct 30 20:03:02.227416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.227087    4320 scope.go:122] "RemoveContainer" containerID="8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"
Oct 30 20:03:02.228753 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.228728    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"
Oct 30 20:03:02.228869 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.228721    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx" event={"ID":"7325a073-a722-4a52-92de-24cb960845db","Type":"ContainerDied","Data":"f8e7f4c6eae01910af987b454ba2120b7d094f3ebb31c2bc3ec4b23cc231e453"}
Oct 30 20:03:02.236519 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.236496    4320 scope.go:122] "RemoveContainer" containerID="8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"
Oct 30 20:03:02.236824 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:02.236805    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6\": container with ID starting with 8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6 not found: ID does not exist" containerID="8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"
Oct 30 20:03:02.236895 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.236834    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6"} err="failed to get container status \"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6\": rpc error: code = NotFound desc = could not find container \"8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6\": container with ID starting with 8c5fef0779eb55910b6f2aae534f6098a93424dd548b2ae8af68c26b9b907ec6 not found: ID does not exist"
Oct 30 20:03:02.236895 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.236852    4320 scope.go:122] "RemoveContainer" containerID="837790640aa43186118e290c921d7b76ce0b022f97ecf640f3aff7ef699d1b20"
Oct 30 20:03:02.278671 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.278647    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 20:03:02.286539 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.286517    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9eb76-85c67d77b-j4jxb"]
Oct 30 20:03:02.324674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.324652    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 20:03:02.328768 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:02.328749    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c5449c-7ccd5b69b8-qlbfx"]
Oct 30 20:03:03.157003 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:03.156970    4320 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:03:03.157185 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:03.157024    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:03:04.096749 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:04.096706    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7325a073-a722-4a52-92de-24cb960845db" path="/var/lib/kubelet/pods/7325a073-a722-4a52-92de-24cb960845db/volumes"
Oct 30 20:03:04.097294 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:04.097272    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" path="/var/lib/kubelet/pods/b06ceee5-3a98-4928-ac9f-e7e9cadeebca/volumes"
Oct 30 20:03:05.076365 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.076342    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 20:03:05.173335 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.173311    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\" (UniqueName: \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\") pod \"e2bc7621-6bef-42ca-a871-a96b5bac5269\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") "
Oct 30 20:03:05.173765 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.173490    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\") pod \"e2bc7621-6bef-42ca-a871-a96b5bac5269\" (UID: \"e2bc7621-6bef-42ca-a871-a96b5bac5269\") "
Oct 30 20:03:05.173838 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.173817    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume" pod "e2bc7621-6bef-42ca-a871-a96b5bac5269" (UID: "e2bc7621-6bef-42ca-a871-a96b5bac5269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:05.173939 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.173926    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bc7621-6bef-42ca-a871-a96b5bac5269-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:05.175420 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.175401    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92" pod "e2bc7621-6bef-42ca-a871-a96b5bac5269" (UID: "e2bc7621-6bef-42ca-a871-a96b5bac5269"). InnerVolumeSpecName "kube-api-access-9lj92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:05.248565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.248534    4320 generic.go:356] "Generic (PLEG): container finished" podID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerID="3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003" exitCode=0
Oct 30 20:03:05.248714 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.248580    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"
Oct 30 20:03:05.248714 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.248642    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" event={"ID":"e2bc7621-6bef-42ca-a871-a96b5bac5269","Type":"ContainerDied","Data":"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"}
Oct 30 20:03:05.248714 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.248705    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx" event={"ID":"e2bc7621-6bef-42ca-a871-a96b5bac5269","Type":"ContainerDied","Data":"b97bba6d5f60e1c679e5190741f7978f3d7328ec11b52700ba922eccaaecec05"}
Oct 30 20:03:05.248852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.248731    4320 scope.go:122] "RemoveContainer" containerID="3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"
Oct 30 20:03:05.258363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.258347    4320 scope.go:122] "RemoveContainer" containerID="3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"
Oct 30 20:03:05.258648 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:05.258629    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003\": container with ID starting with 3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003 not found: ID does not exist" containerID="3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"
Oct 30 20:03:05.258706 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.258656    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003"} err="failed to get container status \"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003\": rpc error: code = NotFound desc = could not find container \"3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003\": container with ID starting with 3d2ef30b18025efc664079f2260cfc992cc645c3bc52536b77d97918f068e003 not found: ID does not exist"
Oct 30 20:03:05.275095 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.275070    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lj92\" (UniqueName: \"kubernetes.io/projected/e2bc7621-6bef-42ca-a871-a96b5bac5269-kube-api-access-9lj92\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:05.303795 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.303771    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 20:03:05.305950 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:05.305927    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c5449c-7fcfcf44c5-658vx"]
Oct 30 20:03:06.100653 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.100588    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" path="/var/lib/kubelet/pods/e2bc7621-6bef-42ca-a871-a96b5bac5269/volumes"
Oct 30 20:03:06.888459 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.888434    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 20:03:06.990989 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.990915    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.990989 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.990956    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.990989 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.990987    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991023    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991062    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991113    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991221    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991266    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991504 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991300    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\") pod \"409bc3b1-677f-4e5e-ac36-f460e734c49c\" (UID: \"409bc3b1-677f-4e5e-ac36-f460e734c49c\") "
Oct 30 20:03:06.991676 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991650    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:06.991914 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991888    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:06.991983 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.991918    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409bc3b1-677f-4e5e-ac36-f460e734c49c-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:06.992150 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.992127    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:06.992322 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.992303    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:06.992406 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.992387    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:06.993493 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.993467    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:06.993724 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.993691    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:06.993870 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.993742    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:06.994906 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:06.994883    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv" pod "409bc3b1-677f-4e5e-ac36-f460e734c49c" (UID: "409bc3b1-677f-4e5e-ac36-f460e734c49c"). InnerVolumeSpecName "kube-api-access-b74bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:07.077951 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.077924    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 20:03:07.082773 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.082748    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 20:03:07.091525 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.091495    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 20:03:07.094070 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094044    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094073    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094089    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094102    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094116    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094130    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74bv\" (UniqueName: \"kubernetes.io/projected/409bc3b1-677f-4e5e-ac36-f460e734c49c-kube-api-access-b74bv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094143    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.094257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.094157    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409bc3b1-677f-4e5e-ac36-f460e734c49c-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:07.104315 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.104293    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 20:03:07.275800 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.275704    4320 generic.go:356] "Generic (PLEG): container finished" podID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerID="ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6" exitCode=0
Oct 30 20:03:07.275800 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.275787    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" event={"ID":"409bc3b1-677f-4e5e-ac36-f460e734c49c","Type":"ContainerDied","Data":"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"}
Oct 30 20:03:07.276054 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.275841    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv" event={"ID":"409bc3b1-677f-4e5e-ac36-f460e734c49c","Type":"ContainerDied","Data":"b6181fdac6fb7f753547249444ee69140551fbe23f5a375684b6713b39d3ea5f"}
Oct 30 20:03:07.276054 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.275870    4320 scope.go:122] "RemoveContainer" containerID="ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"
Oct 30 20:03:07.276054 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.276019    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"
Oct 30 20:03:07.288929 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.288905    4320 scope.go:122] "RemoveContainer" containerID="ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"
Oct 30 20:03:07.289202 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:07.289181    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6\": container with ID starting with ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6 not found: ID does not exist" containerID="ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"
Oct 30 20:03:07.289282 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.289217    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6"} err="failed to get container status \"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6\": rpc error: code = NotFound desc = could not find container \"ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6\": container with ID starting with ee948c5844c211e78d52ea229e4ec73a6631350ccdc7f493244c9e73977b12f6 not found: ID does not exist"
Oct 30 20:03:07.327065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.327039    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 20:03:07.330571 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:07.330547    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5ggbv"]
Oct 30 20:03:08.097357 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:08.097322    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" path="/var/lib/kubelet/pods/409bc3b1-677f-4e5e-ac36-f460e734c49c/volumes"
Oct 30 20:03:09.341401 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:09.327048    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 20:03:09.341401 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:09.327271    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy" containerID="cri-o://8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f" gracePeriod=30
Oct 30 20:03:14.595001 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.594973    4320 scope.go:122] "RemoveContainer" containerID="dde9942a5771d989749f67a667aa9cbe2754aacc05604da9d0d29577606dc212"
Oct 30 20:03:14.620498 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.620471    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 20:03:14.711056 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711024    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711110    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711184    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711337 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711231    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711337 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711257    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711337 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711311    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711379    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711431    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711310    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:14.711502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711459    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\") pod \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\" (UID: \"a546609d-ac82-4bf7-816c-ec2e4c7ce11a\") "
Oct 30 20:03:14.711502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711439    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:14.711815 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711537    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:14.711920 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711827    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:14.711920 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711835    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:14.712029 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711972    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.712029 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.711990    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.712029 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.712004    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.712029 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.712017    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.712176 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.712033    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.713511 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.713481    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:14.713630 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.713486    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:14.713813 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.713793    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:14.713868 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.713809    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7" pod "a546609d-ac82-4bf7-816c-ec2e4c7ce11a" (UID: "a546609d-ac82-4bf7-816c-ec2e4c7ce11a"). InnerVolumeSpecName "kube-api-access-8v2x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:14.813062 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.813037    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2x7\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-kube-api-access-8v2x7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.813062 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.813060    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.813202 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.813069    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:14.813202 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:14.813078    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a546609d-ac82-4bf7-816c-ec2e4c7ce11a-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:15.334009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.333973    4320 generic.go:356] "Generic (PLEG): container finished" podID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerID="8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f" exitCode=0
Oct 30 20:03:15.334213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.334030    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"
Oct 30 20:03:15.334213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.334058    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" event={"ID":"a546609d-ac82-4bf7-816c-ec2e4c7ce11a","Type":"ContainerDied","Data":"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"}
Oct 30 20:03:15.334213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.334108    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6" event={"ID":"a546609d-ac82-4bf7-816c-ec2e4c7ce11a","Type":"ContainerDied","Data":"0c9ced4f691102242bef6d5e2657c76f5b1d87601f54c44cea9ee95eef579755"}
Oct 30 20:03:15.334213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.334136    4320 scope.go:122] "RemoveContainer" containerID="8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"
Oct 30 20:03:15.344043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.344024    4320 scope.go:122] "RemoveContainer" containerID="8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"
Oct 30 20:03:15.344334 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:15.344310    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f\": container with ID starting with 8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f not found: ID does not exist" containerID="8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"
Oct 30 20:03:15.344438 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.344338    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f"} err="failed to get container status \"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f\": rpc error: code = NotFound desc = could not find container \"8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f\": container with ID starting with 8af6a27974ba5b81f55a70ee92b5d58666abb68b3f451558337a9a9bb6cce05f not found: ID does not exist"
Oct 30 20:03:15.395977 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.395946    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 20:03:15.407735 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:15.407713    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-j76b6"]
Oct 30 20:03:16.096228 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.096193    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" path="/var/lib/kubelet/pods/a546609d-ac82-4bf7-816c-ec2e4c7ce11a/volumes"
Oct 30 20:03:16.111300 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111271    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"]
Oct 30 20:03:16.111764 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111749    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111768    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111786    4320 cpu_manager.go:414] "containerMap: removing container" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111793    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111803    4320 cpu_manager.go:414] "containerMap: removing container" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111808    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing"
Oct 30 20:03:16.111827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111824    4320 cpu_manager.go:414] "containerMap: removing container" podUID="7325a073-a722-4a52-92de-24cb960845db" containerName="maas-api"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111831    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7325a073-a722-4a52-92de-24cb960845db" containerName="maas-api"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111841    4320 cpu_manager.go:414] "containerMap: removing container" podUID="80984c1d-f330-4004-8086-d143a6a6e93e" containerName="revoke-keys"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111850    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80984c1d-f330-4004-8086-d143a6a6e93e" containerName="revoke-keys"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111868    4320 cpu_manager.go:414] "containerMap: removing container" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111875    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111887    4320 cpu_manager.go:414] "containerMap: removing container" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111892    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111901    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111906    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111914    4320 cpu_manager.go:414] "containerMap: removing container" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111919    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111940    4320 cpu_manager.go:414] "containerMap: removing container" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111945    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff18f27f-6755-4e38-b9cc-2879a4d17529" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111953    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111957    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111967    4320 cpu_manager.go:414] "containerMap: removing container" podUID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" containerName="maas-api"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111971    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" containerName="maas-api"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111977    4320 cpu_manager.go:414] "containerMap: removing container" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.111982    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cdbcb2-e423-4694-8b14-9a0c6ae2e987" containerName="cleanup"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112051    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c11a744-2060-412d-9f50-62a2b28bb993" containerName="main"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112059    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="a546609d-ac82-4bf7-816c-ec2e4c7ce11a" containerName="istio-proxy"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112067    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ebcd68-f7db-4238-87db-38b0c1b83d3a" containerName="cleanup"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112074    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="414ca6a0-ed48-4c50-851d-8e2063ceed6a" containerName="maas-api"
Oct 30 20:03:16.112076 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112081    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b259aa-f4d5-46c0-be7e-3b95c53e13ae" containerName="cleanup"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112090    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bc7621-6bef-42ca-a871-a96b5bac5269" containerName="payload-pre-processing"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112102    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06ceee5-3a98-4928-ac9f-e7e9cadeebca" containerName="payload-processing"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112113    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9353dd8-420e-469b-9c85-429d115dd184" containerName="istio-proxy"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112119    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="7325a073-a722-4a52-92de-24cb960845db" containerName="maas-api"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112125    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="80984c1d-f330-4004-8086-d143a6a6e93e" containerName="revoke-keys"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112131    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="409bc3b1-677f-4e5e-ac36-f460e734c49c" containerName="istio-proxy"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112137    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef1776d-7576-4554-94e0-30b201c7009c" containerName="cleanup"
Oct 30 20:03:16.112803 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.112143    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="f881ead9-5adb-443c-aabb-8792af9e5cca" containerName="cleanup"
Oct 30 20:03:16.117235 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.117220    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.183462 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.183427    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:03:16.183808 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.183785    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:16.184087 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.184070    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:03:16.185109 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.185071    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:03:16.203214 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.202676    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"]
Oct 30 20:03:16.240401 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.240372    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkfx\" (UniqueName: \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.240535 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.240486    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.240573 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.240542    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.341964 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.341931    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkfx\" (UniqueName: \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.342143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.342020    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.342143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.342082    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.342783 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.342759    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.344416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.344396    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.362479 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.362422    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkfx\" (UniqueName: \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\") pod \"maas-api-revoke-sub-70b8daa9-75d7d\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.432219 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.432193    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:16.738772 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:16.738737    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"]
Oct 30 20:03:16.753101 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:03:16.753066    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b81ca0d_2bd1_4762_85f9_a9288645edbb.slice/crio-a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09 WatchSource:0}: Error finding container a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09: Status 404 returned error can't find the container with id a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09
Oct 30 20:03:17.348974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:17.348883    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d" event={"ID":"5b81ca0d-2bd1-4762-85f9-a9288645edbb","Type":"ContainerStarted","Data":"1eb4aec036d11f4008fe4d1fe1c760e8004a564d0a99f556c6294ccf2905bd5f"}
Oct 30 20:03:17.348974 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:17.348949    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d" event={"ID":"5b81ca0d-2bd1-4762-85f9-a9288645edbb","Type":"ContainerStarted","Data":"a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09"}
Oct 30 20:03:17.393064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:17.392996    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d" podStartSLOduration=1.3929791040000001 podStartE2EDuration="1.392979104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:17.391625239 +0000 UTC m=+3545.427137701" watchObservedRunningTime="2026-10-30 20:03:17.392979104 +0000 UTC m=+3545.428491571"
Oct 30 20:03:18.356062 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:18.356027    4320 generic.go:356] "Generic (PLEG): container finished" podID="5b81ca0d-2bd1-4762-85f9-a9288645edbb" containerID="1eb4aec036d11f4008fe4d1fe1c760e8004a564d0a99f556c6294ccf2905bd5f" exitCode=0
Oct 30 20:03:18.356457 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:18.356090    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d" event={"ID":"5b81ca0d-2bd1-4762-85f9-a9288645edbb","Type":"ContainerDied","Data":"1eb4aec036d11f4008fe4d1fe1c760e8004a564d0a99f556c6294ccf2905bd5f"}
Oct 30 20:03:19.500244 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.500221    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:19.676494 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.676464    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\" (UniqueName: \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\") pod \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") "
Oct 30 20:03:19.676680 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.676565    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\") pod \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") "
Oct 30 20:03:19.676680 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.676674    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\") pod \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\" (UID: \"5b81ca0d-2bd1-4762-85f9-a9288645edbb\") "
Oct 30 20:03:19.676998 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.676976    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca" pod "5b81ca0d-2bd1-4762-85f9-a9288645edbb" (UID: "5b81ca0d-2bd1-4762-85f9-a9288645edbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:19.678621 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.678577    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca" pod "5b81ca0d-2bd1-4762-85f9-a9288645edbb" (UID: "5b81ca0d-2bd1-4762-85f9-a9288645edbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:19.678700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.678581    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx" pod "5b81ca0d-2bd1-4762-85f9-a9288645edbb" (UID: "5b81ca0d-2bd1-4762-85f9-a9288645edbb"). InnerVolumeSpecName "kube-api-access-6mkfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:19.778331 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.778308    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:19.778331 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.778332    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mkfx\" (UniqueName: \"kubernetes.io/projected/5b81ca0d-2bd1-4762-85f9-a9288645edbb-kube-api-access-6mkfx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:19.778458 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:19.778342    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b81ca0d-2bd1-4762-85f9-a9288645edbb-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:20.367973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:20.367894    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d" event={"ID":"5b81ca0d-2bd1-4762-85f9-a9288645edbb","Type":"ContainerDied","Data":"a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09"}
Oct 30 20:03:20.367973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:20.367928    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"
Oct 30 20:03:20.367973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:20.367949    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ad22409e6a66c16c31a308e49880920e8813247fa6c70cb615792d53fd6d09"
Oct 30 20:03:21.982903 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:21.982867    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:03:22.103721 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.103682    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" containerID="cri-o://fbc3b0c8c5968e0905406b33d0c40b5bf60697ce94c7a54716062bcba2269fd6" gracePeriod=45
Oct 30 20:03:22.105183 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.105149    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" containerID="cri-o://a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0" gracePeriod=45
Oct 30 20:03:22.116003 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.115960    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main" containerID="cri-o://ead3ca977c59bb201e4e4e70000f51820060b8bb116b20191334386227578406" gracePeriod=45
Oct 30 20:03:22.126095 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.126066    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" containerID="cri-o://4d88084238ba3d53fe9c3034154cd47aa7dfda902229e8f97c77fa5ec948399a" gracePeriod=45
Oct 30 20:03:22.184768 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.184730    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Oct 30 20:03:22.375502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.375468    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 20:03:22.383342 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.383318    4320 generic.go:356] "Generic (PLEG): container finished" podID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerID="ead3ca977c59bb201e4e4e70000f51820060b8bb116b20191334386227578406" exitCode=0
Oct 30 20:03:22.383483 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.383403    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" event={"ID":"d9e6bd84-7550-4aa1-b304-567e34d5c462","Type":"ContainerDied","Data":"ead3ca977c59bb201e4e4e70000f51820060b8bb116b20191334386227578406"}
Oct 30 20:03:22.385363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.385337    4320 generic.go:356] "Generic (PLEG): container finished" podID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerID="fbc3b0c8c5968e0905406b33d0c40b5bf60697ce94c7a54716062bcba2269fd6" exitCode=0
Oct 30 20:03:22.385498 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.385410    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" event={"ID":"828a1b40-db68-403d-a73d-e8ffc6b31e8f","Type":"ContainerDied","Data":"fbc3b0c8c5968e0905406b33d0c40b5bf60697ce94c7a54716062bcba2269fd6"}
Oct 30 20:03:22.386983 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.386964    4320 generic.go:356] "Generic (PLEG): container finished" podID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerID="a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0" exitCode=0
Oct 30 20:03:22.387218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.387185    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" event={"ID":"fec8e3f9-0c45-4a84-b703-430f061ba2f2","Type":"ContainerDied","Data":"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"}
Oct 30 20:03:22.387218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.387130    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"
Oct 30 20:03:22.387584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.387561    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" event={"ID":"fec8e3f9-0c45-4a84-b703-430f061ba2f2","Type":"ContainerDied","Data":"cfde0de1af74919bc73917b97f923e8bd9b73f79f77e7a6ba77845336ee6518e"}
Oct 30 20:03:22.387703 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.387616    4320 scope.go:122] "RemoveContainer" containerID="a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"
Oct 30 20:03:22.389216 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.389189    4320 generic.go:356] "Generic (PLEG): container finished" podID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerID="4d88084238ba3d53fe9c3034154cd47aa7dfda902229e8f97c77fa5ec948399a" exitCode=0
Oct 30 20:03:22.389324 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.389258    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" event={"ID":"dd4fbb1b-5826-43f7-b333-2e2365ab305f","Type":"ContainerDied","Data":"4d88084238ba3d53fe9c3034154cd47aa7dfda902229e8f97c77fa5ec948399a"}
Oct 30 20:03:22.398100 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.398078    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 20:03:22.399185 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.399170    4320 scope.go:122] "RemoveContainer" containerID="a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"
Oct 30 20:03:22.399419 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:22.399401    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0\": container with ID starting with a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0 not found: ID does not exist" containerID="a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"
Oct 30 20:03:22.399480 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.399426    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0"} err="failed to get container status \"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0\": rpc error: code = NotFound desc = could not find container \"a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0\": container with ID starting with a24a9d86abd7b5bbec1cc9c953d0fb7a6fa0a5e111214d35fdf5611dd060ebc0 not found: ID does not exist"
Oct 30 20:03:22.439287 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.439267    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 20:03:22.444000 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.443982    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 20:03:22.504696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504616    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.504867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504698    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.504867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504731    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.504867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504761    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.504867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504785    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.504867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504834    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504895    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\" (UniqueName: \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504939    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.504965    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505012    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505026    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505038    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505094    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.505180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505157    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\" (UniqueName: \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\") pod \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\" (UID: \"fec8e3f9-0c45-4a84-b703-430f061ba2f2\") "
Oct 30 20:03:22.505685 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505255    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\") pod \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\" (UID: \"828a1b40-db68-403d-a73d-e8ffc6b31e8f\") "
Oct 30 20:03:22.505685 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505279    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.505685 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505331    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.505685 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.505498    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506184    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506217    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506232    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506247    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506261    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.506303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.506273    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.509061 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509034    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:22.509205 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509167    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:22.509205 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509182    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.509575 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509524    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.509683 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509614    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg" pod "fec8e3f9-0c45-4a84-b703-430f061ba2f2" (UID: "fec8e3f9-0c45-4a84-b703-430f061ba2f2"). InnerVolumeSpecName "kube-api-access-89jfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:22.509683 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.509638    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg" pod "828a1b40-db68-403d-a73d-e8ffc6b31e8f" (UID: "828a1b40-db68-403d-a73d-e8ffc6b31e8f"). InnerVolumeSpecName "kube-api-access-gfzvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:22.607615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607574    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607642    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607683    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607707    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607750    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607781    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\" (UniqueName: \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.607805 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607802    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\" (UniqueName: \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607814    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607825    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607950    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607984    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.607990    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608012    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608023    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\") pod \"d9e6bd84-7550-4aa1-b304-567e34d5c462\" (UID: \"d9e6bd84-7550-4aa1-b304-567e34d5c462\") "
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608110    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608112    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\") pod \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\" (UID: \"dd4fbb1b-5826-43f7-b333-2e2365ab305f\") "
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608288    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608490    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608685    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8e3f9-0c45-4a84-b703-430f061ba2f2-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608706    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608721    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89jfg\" (UniqueName: \"kubernetes.io/projected/fec8e3f9-0c45-4a84-b703-430f061ba2f2-kube-api-access-89jfg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608742    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828a1b40-db68-403d-a73d-e8ffc6b31e8f-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608756    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608769    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.608788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608783    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828a1b40-db68-403d-a73d-e8ffc6b31e8f-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.609296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608797    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.609296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608809    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.609296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608821    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.609296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608834    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec8e3f9-0c45-4a84-b703-430f061ba2f2-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.609296 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.608847    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzvg\" (UniqueName: \"kubernetes.io/projected/828a1b40-db68-403d-a73d-e8ffc6b31e8f-kube-api-access-gfzvg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.610033 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610009    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "kube-api-access-r4q95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:22.610232 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610209    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.610322 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610306    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:22.610454 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610436    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs" pod "dd4fbb1b-5826-43f7-b333-2e2365ab305f" (UID: "dd4fbb1b-5826-43f7-b333-2e2365ab305f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:22.610502 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610472    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:22.610703 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.610681    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j" pod "d9e6bd84-7550-4aa1-b304-567e34d5c462" (UID: "d9e6bd84-7550-4aa1-b304-567e34d5c462"). InnerVolumeSpecName "kube-api-access-qtc5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:22.711336 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711291    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd4fbb1b-5826-43f7-b333-2e2365ab305f-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.711336 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711327    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4q95\" (UniqueName: \"kubernetes.io/projected/dd4fbb1b-5826-43f7-b333-2e2365ab305f-kube-api-access-r4q95\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.711565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711345    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtc5j\" (UniqueName: \"kubernetes.io/projected/d9e6bd84-7550-4aa1-b304-567e34d5c462-kube-api-access-qtc5j\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.711565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711358    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd4fbb1b-5826-43f7-b333-2e2365ab305f-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.711565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711372    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e6bd84-7550-4aa1-b304-567e34d5c462-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.711565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.711388    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e6bd84-7550-4aa1-b304-567e34d5c462-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:22.743552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.743509    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 20:03:22.756696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:22.756628    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx"]
Oct 30 20:03:23.306778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.306737    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-w2-9efc72-kserve-58fddb74b7-xxdpx" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:03:23.310937 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.310897    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:03:23.396645 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.396589    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"
Oct 30 20:03:23.396645 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.396625    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp" event={"ID":"dd4fbb1b-5826-43f7-b333-2e2365ab305f","Type":"ContainerDied","Data":"39e3c45d9e02f49cb17684e3b9ef179cb75a13f71ea6e04844918722a1e9beb2"}
Oct 30 20:03:23.396877 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.396668    4320 scope.go:122] "RemoveContainer" containerID="4d88084238ba3d53fe9c3034154cd47aa7dfda902229e8f97c77fa5ec948399a"
Oct 30 20:03:23.398234 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.398219    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"
Oct 30 20:03:23.398330 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.398244    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj" event={"ID":"d9e6bd84-7550-4aa1-b304-567e34d5c462","Type":"ContainerDied","Data":"b5547fa40afd838b061b1437f34539a75cd07ccd486ad78ed1dfdc52f33a1503"}
Oct 30 20:03:23.399905 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.399886    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs" event={"ID":"828a1b40-db68-403d-a73d-e8ffc6b31e8f","Type":"ContainerDied","Data":"caaae1b8998417583a29ad3d2c5a23e15c10efe185cc8d164254ea380ea4f708"}
Oct 30 20:03:23.399987 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.399911    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"
Oct 30 20:03:23.406874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.406435    4320 scope.go:122] "RemoveContainer" containerID="ead3ca977c59bb201e4e4e70000f51820060b8bb116b20191334386227578406"
Oct 30 20:03:23.415997 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.415966    4320 scope.go:122] "RemoveContainer" containerID="fbc3b0c8c5968e0905406b33d0c40b5bf60697ce94c7a54716062bcba2269fd6"
Oct 30 20:03:23.447517 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.447495    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 20:03:23.450946 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.450928    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/simulator-w2-9efc72-kserve-6b6f5fc67c-9hfqp"]
Oct 30 20:03:23.510783 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.510750    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 20:03:23.514552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.514528    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/distinct-2-w2-9efc72-kserve-5f5dfb8f6d-7j2bs"]
Oct 30 20:03:23.546988 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.546967    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 20:03:23.549862 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:23.549843    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9efc72/premium-w2-9efc72-kserve-59f6db8988-sk5rj"]
Oct 30 20:03:24.095283 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:24.095253    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" path="/var/lib/kubelet/pods/828a1b40-db68-403d-a73d-e8ffc6b31e8f/volumes"
Oct 30 20:03:24.095678 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:24.095662    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" path="/var/lib/kubelet/pods/d9e6bd84-7550-4aa1-b304-567e34d5c462/volumes"
Oct 30 20:03:24.096085 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:24.096070    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" path="/var/lib/kubelet/pods/dd4fbb1b-5826-43f7-b333-2e2365ab305f/volumes"
Oct 30 20:03:24.096772 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:24.096750    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" path="/var/lib/kubelet/pods/fec8e3f9-0c45-4a84-b703-430f061ba2f2/volumes"
Oct 30 20:03:28.643934 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:28.643895    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:03:34.235337 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235306    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"]
Oct 30 20:03:34.235776 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235760    4320 cpu_manager.go:414] "containerMap: removing container" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main"
Oct 30 20:03:34.235776 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235775    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235786    4320 cpu_manager.go:414] "containerMap: removing container" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235795    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235810    4320 cpu_manager.go:414] "containerMap: removing container" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235815    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235827    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5b81ca0d-2bd1-4762-85f9-a9288645edbb" containerName="revoke-keys"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235832    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b81ca0d-2bd1-4762-85f9-a9288645edbb" containerName="revoke-keys"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235847    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main"
Oct 30 20:03:34.235852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235852    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main"
Oct 30 20:03:34.236102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235914    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e6bd84-7550-4aa1-b304-567e34d5c462" containerName="main"
Oct 30 20:03:34.236102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235925    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="828a1b40-db68-403d-a73d-e8ffc6b31e8f" containerName="main"
Oct 30 20:03:34.236102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235932    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b81ca0d-2bd1-4762-85f9-a9288645edbb" containerName="revoke-keys"
Oct 30 20:03:34.236102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235942    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec8e3f9-0c45-4a84-b703-430f061ba2f2" containerName="main"
Oct 30 20:03:34.236102 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.235956    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4fbb1b-5826-43f7-b333-2e2365ab305f" containerName="main"
Oct 30 20:03:34.239180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.239163    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.258021 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.257994    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"]
Oct 30 20:03:34.267708 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.267683    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:03:34.267831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.267751    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:03:34.267831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.267811    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:34.267926 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.267901    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:03:34.326791 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.326749    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.326956 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.326812    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wph2t\" (UniqueName: \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.326956 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.326917    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.427948 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.427922    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.428077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.427967    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wph2t\" (UniqueName: \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.428077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.427991    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.428645 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.428627    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.430288 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.430259    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.446924 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.446897    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wph2t\" (UniqueName: \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.552669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.552592    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:34.847688 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:34.847663    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"]
Oct 30 20:03:34.850592 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:03:34.850562    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5932c771_7210_468f_bb88_da81f3c8b723.slice/crio-7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68 WatchSource:0}: Error finding container 7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68: Status 404 returned error can't find the container with id 7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68
Oct 30 20:03:35.030178 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.030150    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"]
Oct 30 20:03:35.031945 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.031925    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f8ba33-fcpdm"]
Oct 30 20:03:35.264501 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.263524    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:03:35.472678 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.472641    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd" event={"ID":"5932c771-7210-468f-bb88-da81f3c8b723","Type":"ContainerStarted","Data":"ec19d6d0ed979a03d64fa468e1fdab21f51de1936be5bfaa1049ab879deb6a10"}
Oct 30 20:03:35.472881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.472711    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd" event={"ID":"5932c771-7210-468f-bb88-da81f3c8b723","Type":"ContainerStarted","Data":"7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68"}
Oct 30 20:03:35.519555 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:35.519434    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd" podStartSLOduration=1.519414042 podStartE2EDuration="1.519414042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:35.517193438 +0000 UTC m=+3563.552705903" watchObservedRunningTime="2026-10-30 20:03:35.519414042 +0000 UTC m=+3563.554926506"
Oct 30 20:03:36.114627 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:36.097691    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2b0625-77ed-4731-a05e-cd80e2b1dc32" path="/var/lib/kubelet/pods/dc2b0625-77ed-4731-a05e-cd80e2b1dc32/volumes"
Oct 30 20:03:36.482241 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:36.482199    4320 generic.go:356] "Generic (PLEG): container finished" podID="5932c771-7210-468f-bb88-da81f3c8b723" containerID="ec19d6d0ed979a03d64fa468e1fdab21f51de1936be5bfaa1049ab879deb6a10" exitCode=0
Oct 30 20:03:36.482778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:36.482264    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd" event={"ID":"5932c771-7210-468f-bb88-da81f3c8b723","Type":"ContainerDied","Data":"ec19d6d0ed979a03d64fa468e1fdab21f51de1936be5bfaa1049ab879deb6a10"}
Oct 30 20:03:37.623270 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.623248    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:37.768699 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.768625    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\") pod \"5932c771-7210-468f-bb88-da81f3c8b723\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") "
Oct 30 20:03:37.768699 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.768677    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\") pod \"5932c771-7210-468f-bb88-da81f3c8b723\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") "
Oct 30 20:03:37.768935 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.768709    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\" (UniqueName: \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\") pod \"5932c771-7210-468f-bb88-da81f3c8b723\" (UID: \"5932c771-7210-468f-bb88-da81f3c8b723\") "
Oct 30 20:03:37.769117 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.769087    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca" pod "5932c771-7210-468f-bb88-da81f3c8b723" (UID: "5932c771-7210-468f-bb88-da81f3c8b723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:37.770987 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.770964    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca" pod "5932c771-7210-468f-bb88-da81f3c8b723" (UID: "5932c771-7210-468f-bb88-da81f3c8b723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:37.771129 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.771053    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t" pod "5932c771-7210-468f-bb88-da81f3c8b723" (UID: "5932c771-7210-468f-bb88-da81f3c8b723"). InnerVolumeSpecName "kube-api-access-wph2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:37.869978 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.869947    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:37.869978 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.869975    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5932c771-7210-468f-bb88-da81f3c8b723-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:37.869978 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:37.869984    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wph2t\" (UniqueName: \"kubernetes.io/projected/5932c771-7210-468f-bb88-da81f3c8b723-kube-api-access-wph2t\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:38.327958 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.327929    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"]
Oct 30 20:03:38.328346 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.328334    4320 cpu_manager.go:414] "containerMap: removing container" podUID="5932c771-7210-468f-bb88-da81f3c8b723" containerName="revoke-keys"
Oct 30 20:03:38.328394 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.328348    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5932c771-7210-468f-bb88-da81f3c8b723" containerName="revoke-keys"
Oct 30 20:03:38.328451 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.328442    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="5932c771-7210-468f-bb88-da81f3c8b723" containerName="revoke-keys"
Oct 30 20:03:38.331500 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.331481    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.349982 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.349959    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"]
Oct 30 20:03:38.476970 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.476940    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.477094 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.476984    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.477094 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.477008    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52vsz\" (UniqueName: \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.494140 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.494114    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd" event={"ID":"5932c771-7210-468f-bb88-da81f3c8b723","Type":"ContainerDied","Data":"7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68"}
Oct 30 20:03:38.494242 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.494137    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"
Oct 30 20:03:38.494242 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.494160    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b577d21ce5d625323d574624dd1eb9618ebe63b765e76f924b4d44a94573b68"
Oct 30 20:03:38.578077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.578054    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.578199 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.578087    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52vsz\" (UniqueName: \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.578199 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.578194    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.578884 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.578864    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.580416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.580394    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.596827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.596804    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52vsz\" (UniqueName: \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.644471 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.644452    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:38.941827 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:38.941804    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"]
Oct 30 20:03:38.944207 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:03:38.944174    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efb4e0e_b65c_46c3_bb04_6a19f98b6638.slice/crio-288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493 WatchSource:0}: Error finding container 288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493: Status 404 returned error can't find the container with id 288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493
Oct 30 20:03:39.500632 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:39.500577    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj" event={"ID":"2efb4e0e-b65c-46c3-bb04-6a19f98b6638","Type":"ContainerStarted","Data":"cfc35a3ac6d323a244f813916d730ca8102cd70750f040e4beac00de59dfa0e8"}
Oct 30 20:03:39.500814 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:39.500662    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj" event={"ID":"2efb4e0e-b65c-46c3-bb04-6a19f98b6638","Type":"ContainerStarted","Data":"288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493"}
Oct 30 20:03:39.541915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:39.541734    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj" podStartSLOduration=1.541721448 podStartE2EDuration="1.541721448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:39.541299104 +0000 UTC m=+3567.576811570" watchObservedRunningTime="2026-10-30 20:03:39.541721448 +0000 UTC m=+3567.577233913"
Oct 30 20:03:40.508087 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:40.508042    4320 generic.go:356] "Generic (PLEG): container finished" podID="2efb4e0e-b65c-46c3-bb04-6a19f98b6638" containerID="cfc35a3ac6d323a244f813916d730ca8102cd70750f040e4beac00de59dfa0e8" exitCode=0
Oct 30 20:03:40.508534 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:40.508124    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj" event={"ID":"2efb4e0e-b65c-46c3-bb04-6a19f98b6638","Type":"ContainerDied","Data":"cfc35a3ac6d323a244f813916d730ca8102cd70750f040e4beac00de59dfa0e8"}
Oct 30 20:03:41.652998 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.652973    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:41.707176 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.707152    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\") pod \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") "
Oct 30 20:03:41.707323 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.707259    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\" (UniqueName: \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\") pod \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") "
Oct 30 20:03:41.707323 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.707282    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\") pod \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\" (UID: \"2efb4e0e-b65c-46c3-bb04-6a19f98b6638\") "
Oct 30 20:03:41.707785 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.707753    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca" pod "2efb4e0e-b65c-46c3-bb04-6a19f98b6638" (UID: "2efb4e0e-b65c-46c3-bb04-6a19f98b6638"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:41.709310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.709283    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz" pod "2efb4e0e-b65c-46c3-bb04-6a19f98b6638" (UID: "2efb4e0e-b65c-46c3-bb04-6a19f98b6638"). InnerVolumeSpecName "kube-api-access-52vsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:41.709416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.709293    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca" pod "2efb4e0e-b65c-46c3-bb04-6a19f98b6638" (UID: "2efb4e0e-b65c-46c3-bb04-6a19f98b6638"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:41.808172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.808115    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:41.808172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.808141    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52vsz\" (UniqueName: \"kubernetes.io/projected/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-kube-api-access-52vsz\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:41.808172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:41.808151    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efb4e0e-b65c-46c3-bb04-6a19f98b6638-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:42.520089 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:42.520061    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"
Oct 30 20:03:42.520089 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:42.520081    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj" event={"ID":"2efb4e0e-b65c-46c3-bb04-6a19f98b6638","Type":"ContainerDied","Data":"288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493"}
Oct 30 20:03:42.520294 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:42.520128    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288ee4b9ead3758233067fb9ba36697834cff6e952cb8f68b9fe667b4186f493"
Oct 30 20:03:43.465010 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:43.464973    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 20:03:43.465448 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:43.465215    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy" containerID="cri-o://39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704" gracePeriod=30
Oct 30 20:03:46.916744 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:46.916710    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"]
Oct 30 20:03:46.928350 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:46.928323    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-870193-ztmpd"]
Oct 30 20:03:48.095349 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.095313    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5932c771-7210-468f-bb88-da81f3c8b723" path="/var/lib/kubelet/pods/5932c771-7210-468f-bb88-da81f3c8b723/volumes"
Oct 30 20:03:48.737024 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.736965    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 20:03:48.883380 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883351    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883398    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883422    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883444    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883496    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883516    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883572 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883532    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883618    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883690    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\") pod \"96044409-2c69-4def-8e3d-7385c86185db\" (UID: \"96044409-2c69-4def-8e3d-7385c86185db\") "
Oct 30 20:03:48.883938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883725    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:48.883938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883738    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:48.883938 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.883725    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884023    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884059    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884138    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884161    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884179    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96044409-2c69-4def-8e3d-7385c86185db-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.884195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884193    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.884529 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.884208    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.885986 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.885952    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:48.886104 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.886026    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:48.886104 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.886035    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:48.886181 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.886103    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg" pod "96044409-2c69-4def-8e3d-7385c86185db" (UID: "96044409-2c69-4def-8e3d-7385c86185db"). InnerVolumeSpecName "kube-api-access-h9szg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:48.985923 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.985889    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96044409-2c69-4def-8e3d-7385c86185db-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.985923 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.985923    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.986122 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.985938    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96044409-2c69-4def-8e3d-7385c86185db-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:48.986122 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:48.985951    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9szg\" (UniqueName: \"kubernetes.io/projected/96044409-2c69-4def-8e3d-7385c86185db-kube-api-access-h9szg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:03:49.430176 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.430140    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:03:49.562801 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.562758    4320 generic.go:356] "Generic (PLEG): container finished" podID="96044409-2c69-4def-8e3d-7385c86185db" containerID="39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704" exitCode=0
Oct 30 20:03:49.562975 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.562813    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"
Oct 30 20:03:49.562975 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.562839    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" event={"ID":"96044409-2c69-4def-8e3d-7385c86185db","Type":"ContainerDied","Data":"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"}
Oct 30 20:03:49.562975 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.562897    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2" event={"ID":"96044409-2c69-4def-8e3d-7385c86185db","Type":"ContainerDied","Data":"2c55c5c9fcb857f8fbc10dcd0a90f0ccaf4018b3efc8f6444f6ca112050a00a8"}
Oct 30 20:03:49.562975 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.562919    4320 scope.go:122] "RemoveContainer" containerID="39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"
Oct 30 20:03:49.573284 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.573265    4320 scope.go:122] "RemoveContainer" containerID="39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"
Oct 30 20:03:49.573528 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:49.573510    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704\": container with ID starting with 39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704 not found: ID does not exist" containerID="39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"
Oct 30 20:03:49.573592 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.573538    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704"} err="failed to get container status \"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704\": rpc error: code = NotFound desc = could not find container \"39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704\": container with ID starting with 39686811fd81172d31cca64e7b39e2b99451a85364878372a99dcde53b718704 not found: ID does not exist"
Oct 30 20:03:49.616232 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.616207    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 20:03:49.628218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:49.628197    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-xqvg2"]
Oct 30 20:03:50.097314 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:50.097281    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96044409-2c69-4def-8e3d-7385c86185db" path="/var/lib/kubelet/pods/96044409-2c69-4def-8e3d-7385c86185db/volumes"
Oct 30 20:03:55.768672 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:55.768641    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"]
Oct 30 20:03:55.780664 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:55.780631    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5a0a17-7b8gj"]
Oct 30 20:03:56.094809 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:03:56.094736    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efb4e0e-b65c-46c3-bb04-6a19f98b6638" path="/var/lib/kubelet/pods/2efb4e0e-b65c-46c3-bb04-6a19f98b6638/volumes"
Oct 30 20:03:58.642863 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:03:58.642828    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:04:01.426479 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:01.426443    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:04:04.259237 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259200    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"]
Oct 30 20:04:04.259688 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259673    4320 cpu_manager.go:414] "containerMap: removing container" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy"
Oct 30 20:04:04.259743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259689    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy"
Oct 30 20:04:04.259743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259713    4320 cpu_manager.go:414] "containerMap: removing container" podUID="2efb4e0e-b65c-46c3-bb04-6a19f98b6638" containerName="revoke-keys"
Oct 30 20:04:04.259743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259719    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efb4e0e-b65c-46c3-bb04-6a19f98b6638" containerName="revoke-keys"
Oct 30 20:04:04.259843 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259793    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="96044409-2c69-4def-8e3d-7385c86185db" containerName="istio-proxy"
Oct 30 20:04:04.259843 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.259804    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efb4e0e-b65c-46c3-bb04-6a19f98b6638" containerName="revoke-keys"
Oct 30 20:04:04.264534 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.264514    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.282601 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.282574    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"]
Oct 30 20:04:04.292058 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.292020    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:04.292198 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.292048    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:04.292198 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.292110    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:04.292331 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.292219    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:04:04.345549 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.345524    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.345704 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.345591    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.345780 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.345720    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56h7l\" (UniqueName: \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.446544 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.446514    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56h7l\" (UniqueName: \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.446711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.446627    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.446791 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.446725    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.447293 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.447274    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.448922 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.448903    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.466341 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.466318    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56h7l\" (UniqueName: \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\") pod \"maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.577816 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.577735    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:04.895315 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:04.895288    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"]
Oct 30 20:04:04.897810 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:04:04.897781    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94202ec5_3494_4bd8_ad7d_23b206662d39.slice/crio-7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e WatchSource:0}: Error finding container 7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e: Status 404 returned error can't find the container with id 7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e
Oct 30 20:04:05.650232 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:05.650195    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq" event={"ID":"94202ec5-3494-4bd8-ad7d-23b206662d39","Type":"ContainerStarted","Data":"54e96a4c435ddb6687120b88a2f077faf42e6ec8a786794a3c2c13d0499ee007"}
Oct 30 20:04:05.650630 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:05.650257    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq" event={"ID":"94202ec5-3494-4bd8-ad7d-23b206662d39","Type":"ContainerStarted","Data":"7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e"}
Oct 30 20:04:05.689987 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:05.689940    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq" podStartSLOduration=1.689926216 podStartE2EDuration="1.689926216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:05.688006546 +0000 UTC m=+3593.723519012" watchObservedRunningTime="2026-10-30 20:04:05.689926216 +0000 UTC m=+3593.725438671"
Oct 30 20:04:06.657225 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:06.657190    4320 generic.go:356] "Generic (PLEG): container finished" podID="94202ec5-3494-4bd8-ad7d-23b206662d39" containerID="54e96a4c435ddb6687120b88a2f077faf42e6ec8a786794a3c2c13d0499ee007" exitCode=0
Oct 30 20:04:06.657652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:06.657281    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq" event={"ID":"94202ec5-3494-4bd8-ad7d-23b206662d39","Type":"ContainerDied","Data":"54e96a4c435ddb6687120b88a2f077faf42e6ec8a786794a3c2c13d0499ee007"}
Oct 30 20:04:07.802869 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.802843    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:07.887251 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.887227    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\") pod \"94202ec5-3494-4bd8-ad7d-23b206662d39\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") "
Oct 30 20:04:07.887399 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.887271    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\" (UniqueName: \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\") pod \"94202ec5-3494-4bd8-ad7d-23b206662d39\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") "
Oct 30 20:04:07.887399 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.887307    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\") pod \"94202ec5-3494-4bd8-ad7d-23b206662d39\" (UID: \"94202ec5-3494-4bd8-ad7d-23b206662d39\") "
Oct 30 20:04:07.887700 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.887675    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca" pod "94202ec5-3494-4bd8-ad7d-23b206662d39" (UID: "94202ec5-3494-4bd8-ad7d-23b206662d39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:07.889187 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.889165    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca" pod "94202ec5-3494-4bd8-ad7d-23b206662d39" (UID: "94202ec5-3494-4bd8-ad7d-23b206662d39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:07.889289 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.889270    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l" pod "94202ec5-3494-4bd8-ad7d-23b206662d39" (UID: "94202ec5-3494-4bd8-ad7d-23b206662d39"). InnerVolumeSpecName "kube-api-access-56h7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:07.988590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.988535    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:07.988590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.988558    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56h7l\" (UniqueName: \"kubernetes.io/projected/94202ec5-3494-4bd8-ad7d-23b206662d39-kube-api-access-56h7l\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:07.988590 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:07.988568    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94202ec5-3494-4bd8-ad7d-23b206662d39-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:08.669074 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:08.669043    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"
Oct 30 20:04:08.669074 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:08.669051    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq" event={"ID":"94202ec5-3494-4bd8-ad7d-23b206662d39","Type":"ContainerDied","Data":"7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e"}
Oct 30 20:04:08.669322 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:08.669099    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b209ec4c4ee24a10e41687bc2163e75536c9ac1fc014e30ab2b7b63b483064e"
Oct 30 20:04:10.002027 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:10.001987    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 20:04:14.664770 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:14.664731    4320 scope.go:122] "RemoveContainer" containerID="5082abce7d728f1239a51fbe464d064fff6ab18ef5fd792b35e101da97c5a007"
Oct 30 20:04:14.676530 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:14.676486    4320 scope.go:122] "RemoveContainer" containerID="d2d762f5ecd7faf173d7ef3e2addfcd98513fe5c48e76e000792d789ca66d114"
Oct 30 20:04:14.788371 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:14.788345    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 20:04:15.517994 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.517957    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"]
Oct 30 20:04:15.518385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.518371    4320 cpu_manager.go:414] "containerMap: removing container" podUID="94202ec5-3494-4bd8-ad7d-23b206662d39" containerName="revoke-keys"
Oct 30 20:04:15.518432 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.518385    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94202ec5-3494-4bd8-ad7d-23b206662d39" containerName="revoke-keys"
Oct 30 20:04:15.518484 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.518478    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="94202ec5-3494-4bd8-ad7d-23b206662d39" containerName="revoke-keys"
Oct 30 20:04:15.523879 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.523853    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.538830 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.538810    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:15.551183 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.551160    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:04:15.554519 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.554491    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"]
Oct 30 20:04:15.566793 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.566771    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:04:15.566874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.566798    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:04:15.573028 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.573006    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 20:04:15.659766 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.659719    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.659956 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.659932    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.660105 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.660082    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2589\" (UniqueName: \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.761205 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.761173    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.761575 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.761218    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.761575 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.761285    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2589\" (UniqueName: \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.761927 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.761909    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.763657 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.763633    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.781235 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.781180    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2589\" (UniqueName: \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\") pod \"maas-api-revoke-sub-3e27f339-rndnd\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:15.837972 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:15.837939    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:16.138206 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:16.138185    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"]
Oct 30 20:04:16.140775 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:04:16.140737    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a1352d_4307_4a85_b613_f4e606497350.slice/crio-49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968 WatchSource:0}: Error finding container 49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968: Status 404 returned error can't find the container with id 49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968
Oct 30 20:04:16.714461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:16.714384    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd" event={"ID":"06a1352d-4307-4a85-b613-f4e606497350","Type":"ContainerStarted","Data":"146f4c81aca1012b314ce11647b39f540f07c8243e9a1fecb96307d30f03c176"}
Oct 30 20:04:16.714461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:16.714434    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd" event={"ID":"06a1352d-4307-4a85-b613-f4e606497350","Type":"ContainerStarted","Data":"49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968"}
Oct 30 20:04:16.756075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:16.756032    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd" podStartSLOduration=1.756018685 podStartE2EDuration="1.756018685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:16.753536459 +0000 UTC m=+3604.789048937" watchObservedRunningTime="2026-10-30 20:04:16.756018685 +0000 UTC m=+3604.791531148"
Oct 30 20:04:17.301149 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.301069    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"]
Oct 30 20:04:17.304919 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.304894    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.328744 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.328713    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"]
Oct 30 20:04:17.379941 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.379913    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.380084 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.379952    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.380084 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.379986    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkwv\" (UniqueName: \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.481606 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.481553    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.481777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.481634    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.481777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.481674    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkwv\" (UniqueName: \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.482258 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.482234    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.484030 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.484007    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.500950 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.500919    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkwv\" (UniqueName: \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\") pod \"maas-api-revoke-sub-6fc11968-lx9zf\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.618579 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.618504    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:17.721900 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.721866    4320 generic.go:356] "Generic (PLEG): container finished" podID="06a1352d-4307-4a85-b613-f4e606497350" containerID="146f4c81aca1012b314ce11647b39f540f07c8243e9a1fecb96307d30f03c176" exitCode=0
Oct 30 20:04:17.722049 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.721943    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd" event={"ID":"06a1352d-4307-4a85-b613-f4e606497350","Type":"ContainerDied","Data":"146f4c81aca1012b314ce11647b39f540f07c8243e9a1fecb96307d30f03c176"}
Oct 30 20:04:17.919882 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:17.919796    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"]
Oct 30 20:04:17.922267 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:04:17.922240    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6009f59a_c864_4c10_bbeb_81b4963ba505.slice/crio-56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2 WatchSource:0}: Error finding container 56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2: Status 404 returned error can't find the container with id 56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2
Oct 30 20:04:18.728379 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.728353    4320 generic.go:356] "Generic (PLEG): container finished" podID="6009f59a-c864-4c10-bbeb-81b4963ba505" containerID="6fe77ff6099134667cde5c1ca0c3873699a3af89959af34b52f5c2719037371e" exitCode=0
Oct 30 20:04:18.728804 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.728442    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf" event={"ID":"6009f59a-c864-4c10-bbeb-81b4963ba505","Type":"ContainerDied","Data":"6fe77ff6099134667cde5c1ca0c3873699a3af89959af34b52f5c2719037371e"}
Oct 30 20:04:18.728804 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.728515    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf" event={"ID":"6009f59a-c864-4c10-bbeb-81b4963ba505","Type":"ContainerStarted","Data":"56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2"}
Oct 30 20:04:18.867855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.867832    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:18.997003 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.996936    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\" (UniqueName: \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\") pod \"06a1352d-4307-4a85-b613-f4e606497350\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") "
Oct 30 20:04:18.997130 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.997034    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\") pod \"06a1352d-4307-4a85-b613-f4e606497350\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") "
Oct 30 20:04:18.997130 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.997090    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\") pod \"06a1352d-4307-4a85-b613-f4e606497350\" (UID: \"06a1352d-4307-4a85-b613-f4e606497350\") "
Oct 30 20:04:18.997387 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.997360    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca" pod "06a1352d-4307-4a85-b613-f4e606497350" (UID: "06a1352d-4307-4a85-b613-f4e606497350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:18.998963 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.998939    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589" pod "06a1352d-4307-4a85-b613-f4e606497350" (UID: "06a1352d-4307-4a85-b613-f4e606497350"). InnerVolumeSpecName "kube-api-access-h2589". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:18.999122 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:18.999101    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca" pod "06a1352d-4307-4a85-b613-f4e606497350" (UID: "06a1352d-4307-4a85-b613-f4e606497350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:19.098652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.098629    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2589\" (UniqueName: \"kubernetes.io/projected/06a1352d-4307-4a85-b613-f4e606497350-kube-api-access-h2589\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:19.098652 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.098648    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a1352d-4307-4a85-b613-f4e606497350-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:19.098787 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.098658    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a1352d-4307-4a85-b613-f4e606497350-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:19.734308 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.734283    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"
Oct 30 20:04:19.734695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.734317    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd" event={"ID":"06a1352d-4307-4a85-b613-f4e606497350","Type":"ContainerDied","Data":"49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968"}
Oct 30 20:04:19.734695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.734362    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ae11eefb31dac2a729116539505d328392f0527dd9662d97bfb5d9c4061968"
Oct 30 20:04:19.789679 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.789635    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing" containerID="cri-o://8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021" gracePeriod=25
Oct 30 20:04:19.873358 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:19.873334    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:20.007409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.007339    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\") pod \"6009f59a-c864-4c10-bbeb-81b4963ba505\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") "
Oct 30 20:04:20.007552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.007426    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\") pod \"6009f59a-c864-4c10-bbeb-81b4963ba505\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") "
Oct 30 20:04:20.007552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.007474    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\" (UniqueName: \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\") pod \"6009f59a-c864-4c10-bbeb-81b4963ba505\" (UID: \"6009f59a-c864-4c10-bbeb-81b4963ba505\") "
Oct 30 20:04:20.007711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.007664    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca" pod "6009f59a-c864-4c10-bbeb-81b4963ba505" (UID: "6009f59a-c864-4c10-bbeb-81b4963ba505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:20.007836 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.007820    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.009465 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.009441    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv" pod "6009f59a-c864-4c10-bbeb-81b4963ba505" (UID: "6009f59a-c864-4c10-bbeb-81b4963ba505"). InnerVolumeSpecName "kube-api-access-ljkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:20.009552 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.009510    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca" pod "6009f59a-c864-4c10-bbeb-81b4963ba505" (UID: "6009f59a-c864-4c10-bbeb-81b4963ba505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:20.108487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.108466    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6009f59a-c864-4c10-bbeb-81b4963ba505-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.108583 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.108487    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkwv\" (UniqueName: \"kubernetes.io/projected/6009f59a-c864-4c10-bbeb-81b4963ba505-kube-api-access-ljkwv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:20.573283 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.573243    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing" containerID="cri-o://3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8" gracePeriod=25
Oct 30 20:04:20.740136 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.740106    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"
Oct 30 20:04:20.740540 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.740139    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf" event={"ID":"6009f59a-c864-4c10-bbeb-81b4963ba505","Type":"ContainerDied","Data":"56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2"}
Oct 30 20:04:20.740540 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:20.740202    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c50cc9bf74ef093aba0beffb15a8cadedddd778b263fde91753c3eb07824c2"
Oct 30 20:04:24.773182 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:24.773144    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:04:25.022591 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.022546    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" containerID="cri-o://d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9" gracePeriod=45
Oct 30 20:04:25.134919 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.134881    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Oct 30 20:04:25.272579 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.272554    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 20:04:25.456696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.456666    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\" (UniqueName: \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.456905 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.456759    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.456905 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.456819    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.457042 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.456909    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.457042 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.456970    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.457042 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457000    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\" (UniqueName: \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\") pod \"183ed1f5-6b56-4b8e-8537-50a952b89d24\" (UID: \"183ed1f5-6b56-4b8e-8537-50a952b89d24\") "
Oct 30 20:04:25.457197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457053    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:25.457197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457094    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:25.457197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457177    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:25.457469 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457446    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.457554 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457476    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.457554 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.457490    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.458962 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.458940    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:25.459272 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.459247    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "kube-api-access-2hjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:25.459357 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.459338    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm" pod "183ed1f5-6b56-4b8e-8537-50a952b89d24" (UID: "183ed1f5-6b56-4b8e-8537-50a952b89d24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:25.558413 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.558378    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183ed1f5-6b56-4b8e-8537-50a952b89d24-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.558413 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.558409    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjnt\" (UniqueName: \"kubernetes.io/projected/183ed1f5-6b56-4b8e-8537-50a952b89d24-kube-api-access-2hjnt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.558413 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.558420    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183ed1f5-6b56-4b8e-8537-50a952b89d24-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:25.765361 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.765287    4320 generic.go:356] "Generic (PLEG): container finished" podID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerID="d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9" exitCode=0
Oct 30 20:04:25.765361 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.765326    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"
Oct 30 20:04:25.765532 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.765370    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" event={"ID":"183ed1f5-6b56-4b8e-8537-50a952b89d24","Type":"ContainerDied","Data":"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"}
Oct 30 20:04:25.765532 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.765410    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh" event={"ID":"183ed1f5-6b56-4b8e-8537-50a952b89d24","Type":"ContainerDied","Data":"f233c8670cc1e68989c345dda2b8b358200250a87da661e68719e3842c651642"}
Oct 30 20:04:25.765532 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.765428    4320 scope.go:122] "RemoveContainer" containerID="d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"
Oct 30 20:04:25.774877 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.774856    4320 scope.go:122] "RemoveContainer" containerID="d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"
Oct 30 20:04:25.775180 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:25.775139    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9\": container with ID starting with d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9 not found: ID does not exist" containerID="d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"
Oct 30 20:04:25.775180 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.775165    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9"} err="failed to get container status \"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9\": rpc error: code = NotFound desc = could not find container \"d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9\": container with ID starting with d2030e024b7a6af30a2d8bbe4dab859bb707010de0790182d02df2e178492da9 not found: ID does not exist"
Oct 30 20:04:25.781195 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.781171    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:04:25.817743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.817721    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 20:04:25.821518 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:25.821500    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b97379/distinct-w4-b97379-kserve-99fb485fd-tljsh"]
Oct 30 20:04:26.094968 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:26.094898    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" path="/var/lib/kubelet/pods/183ed1f5-6b56-4b8e-8537-50a952b89d24/volumes"
Oct 30 20:04:28.643277 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:28.643244    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:04:30.359543 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:30.359507    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:04:34.267619 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:34.267579    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"]
Oct 30 20:04:34.282197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:34.282125    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e67b84-2f5tq"]
Oct 30 20:04:34.773239 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:34.773204    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:04:35.787973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:35.787934    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:04:36.095573 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:36.095501    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94202ec5-3494-4bd8-ad7d-23b206662d39" path="/var/lib/kubelet/pods/94202ec5-3494-4bd8-ad7d-23b206662d39/volumes"
Oct 30 20:04:39.109926 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:39.109893    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 20:04:39.110424 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:39.110108    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy" containerID="cri-o://4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456" gracePeriod=30
Oct 30 20:04:40.047197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.047138    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 20:04:40.122697 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.122668    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\" (UniqueName: \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\") pod \"c0784043-9b3e-45ff-b204-de0a2d3555e0\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") "
Oct 30 20:04:40.123122 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.122745    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\") pod \"c0784043-9b3e-45ff-b204-de0a2d3555e0\" (UID: \"c0784043-9b3e-45ff-b204-de0a2d3555e0\") "
Oct 30 20:04:40.123207 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.123161    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume" pod "c0784043-9b3e-45ff-b204-de0a2d3555e0" (UID: "c0784043-9b3e-45ff-b204-de0a2d3555e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:40.124817 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.124793    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt" pod "c0784043-9b3e-45ff-b204-de0a2d3555e0" (UID: "c0784043-9b3e-45ff-b204-de0a2d3555e0"). InnerVolumeSpecName "kube-api-access-zsgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:40.224359 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.224334    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0784043-9b3e-45ff-b204-de0a2d3555e0-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:40.224359 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.224357    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgrt\" (UniqueName: \"kubernetes.io/projected/c0784043-9b3e-45ff-b204-de0a2d3555e0-kube-api-access-zsgrt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:40.826519 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.826492    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 20:04:40.858191 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.858128    4320 generic.go:356] "Generic (PLEG): container finished" podID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerID="3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8" exitCode=0
Oct 30 20:04:40.858191 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.858173    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"
Oct 30 20:04:40.858365 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.858205    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" event={"ID":"da82e62d-2257-4445-a27c-b95bb9d14e10","Type":"ContainerDied","Data":"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"}
Oct 30 20:04:40.858365 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.858253    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb" event={"ID":"da82e62d-2257-4445-a27c-b95bb9d14e10","Type":"ContainerDied","Data":"0b303fa43a8acba2975a484fa889f75628b747ff64bb642372ce3ee2acaf6774"}
Oct 30 20:04:40.858365 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.858276    4320 scope.go:122] "RemoveContainer" containerID="3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"
Oct 30 20:04:40.859419 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.859398    4320 generic.go:356] "Generic (PLEG): container finished" podID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerID="8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021" exitCode=0
Oct 30 20:04:40.859522 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.859445    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"
Oct 30 20:04:40.859522 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.859476    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" event={"ID":"c0784043-9b3e-45ff-b204-de0a2d3555e0","Type":"ContainerDied","Data":"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"}
Oct 30 20:04:40.859648 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.859519    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct" event={"ID":"c0784043-9b3e-45ff-b204-de0a2d3555e0","Type":"ContainerDied","Data":"1dc64935e744bf674f1520280cf92fae080c6d39d624d9c8fc7ab218bb58760c"}
Oct 30 20:04:40.868199 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.868176    4320 scope.go:122] "RemoveContainer" containerID="3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"
Oct 30 20:04:40.868816 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:40.868578    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8\": container with ID starting with 3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8 not found: ID does not exist" containerID="3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"
Oct 30 20:04:40.868816 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.868647    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8"} err="failed to get container status \"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8\": rpc error: code = NotFound desc = could not find container \"3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8\": container with ID starting with 3f9038b5b8b0bb1ff766bb115ca0ceedcc05ef274950f014c347b1a9bef394a8 not found: ID does not exist"
Oct 30 20:04:40.868816 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.868671    4320 scope.go:122] "RemoveContainer" containerID="8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"
Oct 30 20:04:40.877769 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.877751    4320 scope.go:122] "RemoveContainer" containerID="8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"
Oct 30 20:04:40.878054 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:40.878035    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021\": container with ID starting with 8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021 not found: ID does not exist" containerID="8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"
Oct 30 20:04:40.878101 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.878059    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021"} err="failed to get container status \"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021\": rpc error: code = NotFound desc = could not find container \"8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021\": container with ID starting with 8cc3b05af12a0a1972d5ef63eaf124949bced99d80a25380e062e722af5ce021 not found: ID does not exist"
Oct 30 20:04:40.930874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.930846    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\" (UniqueName: \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\") pod \"da82e62d-2257-4445-a27c-b95bb9d14e10\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") "
Oct 30 20:04:40.931023 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.930983    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\") pod \"da82e62d-2257-4445-a27c-b95bb9d14e10\" (UID: \"da82e62d-2257-4445-a27c-b95bb9d14e10\") "
Oct 30 20:04:40.931423 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.931399    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume" pod "da82e62d-2257-4445-a27c-b95bb9d14e10" (UID: "da82e62d-2257-4445-a27c-b95bb9d14e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:40.933233 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.933203    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8" pod "da82e62d-2257-4445-a27c-b95bb9d14e10" (UID: "da82e62d-2257-4445-a27c-b95bb9d14e10"). InnerVolumeSpecName "kube-api-access-9pgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:40.934527 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.934500    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 20:04:40.945906 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:40.938924    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e67b84-9578b57dd-9fvct"]
Oct 30 20:04:41.032291 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.032262    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pgw8\" (UniqueName: \"kubernetes.io/projected/da82e62d-2257-4445-a27c-b95bb9d14e10-kube-api-access-9pgw8\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:41.032291 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.032290    4320 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da82e62d-2257-4445-a27c-b95bb9d14e10-plugins-config-volume\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:41.151741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.151710    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 20:04:41.152779 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.151926    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy" containerID="cri-o://e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5" gracePeriod=30
Oct 30 20:04:41.229711 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.229684    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 20:04:41.235534 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.235512    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b97379-58647c55b4-rn5cb"]
Oct 30 20:04:41.242564 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:41.242538    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:04:42.095085 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:42.095053    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" path="/var/lib/kubelet/pods/c0784043-9b3e-45ff-b204-de0a2d3555e0/volumes"
Oct 30 20:04:42.095503 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:42.095485    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" path="/var/lib/kubelet/pods/da82e62d-2257-4445-a27c-b95bb9d14e10/volumes"
Oct 30 20:04:44.353915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.353893    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 20:04:44.462994 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.462971    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463024    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463045    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463075    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463147 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463109    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463151    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463219    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463260    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463310    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\") pod \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\" (UID: \"cd34e558-1afb-4b30-a736-f7c9f7f96ac3\") "
Oct 30 20:04:44.463355 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463324    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:44.463681 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463361    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:44.463681 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463482    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:44.463681 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463567    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:44.463849 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463690    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.463849 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463706    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.463849 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463716    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.463849 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463724    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.463849 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.463778    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:44.465167 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.465142    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:44.465472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.465448    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:44.465549 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.465467    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "kube-api-access-l2nkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:44.465549 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.465490    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo" pod "cd34e558-1afb-4b30-a736-f7c9f7f96ac3" (UID: "cd34e558-1afb-4b30-a736-f7c9f7f96ac3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:44.564833 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.564812    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.564833 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.564832    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.564948 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.564841    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.564948 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.564849    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nkr\" (UniqueName: \"kubernetes.io/projected/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-kube-api-access-l2nkr\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.564948 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.564859    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd34e558-1afb-4b30-a736-f7c9f7f96ac3-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:44.882842 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.882807    4320 generic.go:356] "Generic (PLEG): container finished" podID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerID="4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456" exitCode=0
Oct 30 20:04:44.883026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.882864    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"
Oct 30 20:04:44.883026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.882875    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" event={"ID":"cd34e558-1afb-4b30-a736-f7c9f7f96ac3","Type":"ContainerDied","Data":"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"}
Oct 30 20:04:44.883026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.882971    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd" event={"ID":"cd34e558-1afb-4b30-a736-f7c9f7f96ac3","Type":"ContainerDied","Data":"a97520926cbc8aed5c73b7991d6d0bc355be11d8f6ebcde3715a06d8f2fbd389"}
Oct 30 20:04:44.883026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.883003    4320 scope.go:122] "RemoveContainer" containerID="4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"
Oct 30 20:04:44.892518 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.892497    4320 scope.go:122] "RemoveContainer" containerID="4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"
Oct 30 20:04:44.892865 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:44.892845    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456\": container with ID starting with 4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456 not found: ID does not exist" containerID="4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"
Oct 30 20:04:44.892927 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.892873    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456"} err="failed to get container status \"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456\": rpc error: code = NotFound desc = could not find container \"4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456\": container with ID starting with 4b3ae4c54b7f388fa8a0572ae40eb494792dee51d087cce01703010c4b3c6456 not found: ID does not exist"
Oct 30 20:04:44.944982 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.944957    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 20:04:44.956679 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:44.956657    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-f6ljd"]
Oct 30 20:04:46.095666 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.095634    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" path="/var/lib/kubelet/pods/cd34e558-1afb-4b30-a736-f7c9f7f96ac3/volumes"
Oct 30 20:04:46.422431 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.422399    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 20:04:46.484042 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484003    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484230 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484073    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484230 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484133    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484230 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484191    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484231    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484278    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484330    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484362    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.484429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.484408    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\") pod \"d988974a-2335-4851-bb94-d4964feecb3d\" (UID: \"d988974a-2335-4851-bb94-d4964feecb3d\") "
Oct 30 20:04:46.485524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.485230    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:46.486654 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.485991    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:46.486654 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.486181    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:46.486654 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.486366    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:46.487247 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.487204    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:46.488780 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.488729    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:46.489693 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.489651    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "kube-api-access-4xmhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:46.489942 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.489910    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:46.491830 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.491804    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy" pod "d988974a-2335-4851-bb94-d4964feecb3d" (UID: "d988974a-2335-4851-bb94-d4964feecb3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:46.585310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585277    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585299    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585308    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585316    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d988974a-2335-4851-bb94-d4964feecb3d-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585649 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585327    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585649 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585336    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585649 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585344    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d988974a-2335-4851-bb94-d4964feecb3d-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585649 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585354    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmhg\" (UniqueName: \"kubernetes.io/projected/d988974a-2335-4851-bb94-d4964feecb3d-kube-api-access-4xmhg\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.585649 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.585362    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d988974a-2335-4851-bb94-d4964feecb3d-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:04:46.896459 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.896423    4320 generic.go:356] "Generic (PLEG): container finished" podID="d988974a-2335-4851-bb94-d4964feecb3d" containerID="e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5" exitCode=0
Oct 30 20:04:46.896668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.896476    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"
Oct 30 20:04:46.896668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.896519    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" event={"ID":"d988974a-2335-4851-bb94-d4964feecb3d","Type":"ContainerDied","Data":"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"}
Oct 30 20:04:46.896668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.896590    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn" event={"ID":"d988974a-2335-4851-bb94-d4964feecb3d","Type":"ContainerDied","Data":"1feaf6eaf7e107d3aed14261d8c11c4638b39c405141746cb05289fd66e540b9"}
Oct 30 20:04:46.896668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.896634    4320 scope.go:122] "RemoveContainer" containerID="e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"
Oct 30 20:04:46.909779 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.909751    4320 scope.go:122] "RemoveContainer" containerID="e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"
Oct 30 20:04:46.910095 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:46.910070    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5\": container with ID starting with e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5 not found: ID does not exist" containerID="e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"
Oct 30 20:04:46.910197 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.910108    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5"} err="failed to get container status \"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5\": rpc error: code = NotFound desc = could not find container \"e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5\": container with ID starting with e9c2b7055ad318ea7d628c2a1954c92ffda05e973613815655c5a7311a3cd4c5 not found: ID does not exist"
Oct 30 20:04:46.947772 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.947749    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 20:04:46.959542 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:46.959515    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-f4bxn"]
Oct 30 20:04:48.095677 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:04:48.095645    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d988974a-2335-4851-bb94-d4964feecb3d" path="/var/lib/kubelet/pods/d988974a-2335-4851-bb94-d4964feecb3d/volumes"
Oct 30 20:04:58.643320 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:04:58.643236    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:05:14.723462 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:05:14.723430    4320 scope.go:122] "RemoveContainer" containerID="3efa1970c3e5ab4630b228e1babbbe4f87f79997c0c61c08f6489147525b3f64"
Oct 30 20:05:28.648568 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:05:28.648519    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:05:50.810806 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:05:50.810774    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:05:58.645835 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:05:58.645797    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:06:08.027328 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:08.027280    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:06:10.817126 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817090    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:06:10.817672 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817653    4320 cpu_manager.go:414] "containerMap: removing container" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817673    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817707    4320 cpu_manager.go:414] "containerMap: removing container" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817716    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817731    4320 cpu_manager.go:414] "containerMap: removing container" podUID="6009f59a-c864-4c10-bbeb-81b4963ba505" containerName="revoke-keys"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817740    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6009f59a-c864-4c10-bbeb-81b4963ba505" containerName="revoke-keys"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817757    4320 cpu_manager.go:414] "containerMap: removing container" podUID="06a1352d-4307-4a85-b613-f4e606497350" containerName="revoke-keys"
Oct 30 20:06:10.817774 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817764    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a1352d-4307-4a85-b613-f4e606497350" containerName="revoke-keys"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817784    4320 cpu_manager.go:414] "containerMap: removing container" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817791    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817807    4320 cpu_manager.go:414] "containerMap: removing container" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817814    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817834    4320 cpu_manager.go:414] "containerMap: removing container" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817841    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817927    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="d988974a-2335-4851-bb94-d4964feecb3d" containerName="istio-proxy"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817942    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="183ed1f5-6b56-4b8e-8537-50a952b89d24" containerName="main"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817955    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0784043-9b3e-45ff-b204-de0a2d3555e0" containerName="payload-processing"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817966    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="da82e62d-2257-4445-a27c-b95bb9d14e10" containerName="payload-processing"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817979    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd34e558-1afb-4b30-a736-f7c9f7f96ac3" containerName="istio-proxy"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817988    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="6009f59a-c864-4c10-bbeb-81b4963ba505" containerName="revoke-keys"
Oct 30 20:06:10.818152 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.817997    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a1352d-4307-4a85-b613-f4e606497350" containerName="revoke-keys"
Oct 30 20:06:10.821497 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.821476    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.856713 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.856687    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:06:10.856852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.856756    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:06:10.860736 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.860717    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.891071 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.891047    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:06:10.899213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899191    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899324 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899218    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899324 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899239    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899324 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899261    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899450 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899399    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899497 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899481    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899543 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899507    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899642 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899622    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899712 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899690    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899759 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899743    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b899r\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899796 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899778    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899843 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899808    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9czh7\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.899887 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899865    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899932 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899912    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899984 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899934    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:10.899984 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899951    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.900064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.899995    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:10.900064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:10.900018    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.000926 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.000900    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.000937    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.000967    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.000984    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b899r\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001008    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001207 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001166    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9czh7\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001219    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001254    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001287    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001306    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001319    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001353    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001373 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001355    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001392    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001424    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001452    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001484    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001521    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001560    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001570    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.001698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001641    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.002145 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001813    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.002145 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001836    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.002145 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.001959    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.002273 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.002140    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.002273 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.002144    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.002273 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.002227    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.002461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.002434    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.003769 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.003747    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.004092 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.004066    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.004247 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.004226    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.004338 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.004242    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.025163 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.025144    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.027178 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.027153    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.027690 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.027665    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b899r\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\") pod \"maas-default-gateway-openshift-default-6f54779779-nv972\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.027965 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.027949    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9czh7\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\") pod \"maas-default-gateway-openshift-default-6f54779779-d8qbx\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.134966 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.134941    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:11.173363 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.173337    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:11.452005 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.451981    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:06:11.454278 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:06:11.454249    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9d0999_76b8_45bd_87db_fd086b0ad274.slice/crio-f54630ebc7137f29cf12640513cb8157d18cd523f2a8e1276e599a6661e35831 WatchSource:0}: Error finding container f54630ebc7137f29cf12640513cb8157d18cd523f2a8e1276e599a6661e35831: Status 404 returned error can't find the container with id f54630ebc7137f29cf12640513cb8157d18cd523f2a8e1276e599a6661e35831
Oct 30 20:06:11.456308 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.456276    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:11.456416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.456334    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:11.456416 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.456363    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:11.523853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.523829    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:06:11.526201 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:06:11.526171    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd4fcad_85bf_4edb_b322_fbc4defad9ce.slice/crio-0b3cb012be61d05980b9ff6eb53875e4a797977f9f3459b5ba86ce514ec5bfa8 WatchSource:0}: Error finding container 0b3cb012be61d05980b9ff6eb53875e4a797977f9f3459b5ba86ce514ec5bfa8: Status 404 returned error can't find the container with id 0b3cb012be61d05980b9ff6eb53875e4a797977f9f3459b5ba86ce514ec5bfa8
Oct 30 20:06:11.528301 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.528271    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:11.528394 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.528346    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:11.528461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:11.528391    4320 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:06:12.326752 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.326709    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" event={"ID":"df9d0999-76b8-45bd-87db-fd086b0ad274","Type":"ContainerStarted","Data":"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"}
Oct 30 20:06:12.327213 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.326775    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" event={"ID":"df9d0999-76b8-45bd-87db-fd086b0ad274","Type":"ContainerStarted","Data":"f54630ebc7137f29cf12640513cb8157d18cd523f2a8e1276e599a6661e35831"}
Oct 30 20:06:12.328454 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.328426    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" event={"ID":"fcd4fcad-85bf-4edb-b322-fbc4defad9ce","Type":"ContainerStarted","Data":"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"}
Oct 30 20:06:12.328587 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.328474    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" event={"ID":"fcd4fcad-85bf-4edb-b322-fbc4defad9ce","Type":"ContainerStarted","Data":"0b3cb012be61d05980b9ff6eb53875e4a797977f9f3459b5ba86ce514ec5bfa8"}
Oct 30 20:06:12.394623 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.382145    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podStartSLOduration=2.382126811 podStartE2EDuration="2.382126811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.38104686 +0000 UTC m=+3720.416559325" watchObservedRunningTime="2026-10-30 20:06:12.382126811 +0000 UTC m=+3720.417639279"
Oct 30 20:06:12.442979 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.442927    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podStartSLOduration=2.442911914 podStartE2EDuration="2.442911914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.44010244 +0000 UTC m=+3720.475614903" watchObservedRunningTime="2026-10-30 20:06:12.442911914 +0000 UTC m=+3720.478424378"
Oct 30 20:06:12.940207 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:12.940172    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:06:13.135662 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.135631    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:13.137009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.136982    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:13.137132 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.137026    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:13.185622 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.176909    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:13.185622 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.178532    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:13.185622 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:13.178621    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:14.138123 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:14.138083    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:14.138668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:14.138143    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:14.189970 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:14.189934    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:14.190225 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:14.190198    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:15.136158 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:15.136117    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:15.136492 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:15.136440    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:15.179264 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:15.179226    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:15.179852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:15.179824    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:16.135908 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:16.135878    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:16.136157 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:16.136134    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:16.189628 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:16.184207    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:16.189628 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:16.184274    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:17.135655 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:17.135620    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:17.135848 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:17.135687    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:17.177968 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:17.177932    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:17.178232 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:17.178204    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:18.135423 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:18.135385    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 20:06:18.135840 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:18.135450    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 20:06:18.173775 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:18.173739    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:06:18.173904 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:18.173794    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 20:06:20.137204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:20.137166    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:20.137674 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:20.137228    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.173926 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:20.173894    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:20.174079 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:20.173958    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:21.178784 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.178748    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:21.179426 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.179395    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:21.216808 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.216779    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:21.217024 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.216992    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:21.826697 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.826665    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:21.827361 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:21.827303    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:22.155462 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.155436    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:06:22.258760 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.258731    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:22.259142 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.258800    4320 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:22.394158 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.394128    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:22.394503 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.394482    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:22.395119 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:22.395103    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:06:28.644278 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:06:28.644234    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:06:31.151904 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:31.151869    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:06:40.868936 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:40.868891    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 20:06:40.869390 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:40.869098    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" podUID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" containerName="discovery" containerID="cri-o://eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de" gracePeriod=30
Oct 30 20:06:41.125130 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.125078    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 20:06:41.191713 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191689    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.191853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191726    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.191853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191766    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.191853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191785    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.191853 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191842    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.192100 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191892    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.192100 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.191968    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\") pod \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\" (UID: \"8eae4b64-b77f-46b0-9e14-19b61a39b7d3\") "
Oct 30 20:06:41.192967 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.192935    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:41.194385 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.194347    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:41.194506 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.194449    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:41.194506 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.194458    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "kube-api-access-vbrbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:41.194741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.194724    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:41.194897 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.194878    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:41.195245 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.195219    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token" pod "8eae4b64-b77f-46b0-9e14-19b61a39b7d3" (UID: "8eae4b64-b77f-46b0-9e14-19b61a39b7d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:41.293484 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293453    4320 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-kubeconfig\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293484 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293481    4320 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-cacerts\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293490    4320 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-local-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293500    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrbx\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-kube-api-access-vbrbx\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293508    4320 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-ca-configmap\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293516    4320 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-csr-dns-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.293669 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.293523    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae4b64-b77f-46b0-9e14-19b61a39b7d3-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:06:41.483872 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.483839    4320 generic.go:356] "Generic (PLEG): container finished" podID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" containerID="eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de" exitCode=0
Oct 30 20:06:41.484039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.483882    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"
Oct 30 20:06:41.484039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.483915    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" event={"ID":"8eae4b64-b77f-46b0-9e14-19b61a39b7d3","Type":"ContainerDied","Data":"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"}
Oct 30 20:06:41.484039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.483967    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4" event={"ID":"8eae4b64-b77f-46b0-9e14-19b61a39b7d3","Type":"ContainerDied","Data":"9df94400eb4b877c053023db9e51a69320114320540666dd607fefab000a838d"}
Oct 30 20:06:41.484039 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.483988    4320 scope.go:122] "RemoveContainer" containerID="eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"
Oct 30 20:06:41.496498 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.496479    4320 scope.go:122] "RemoveContainer" containerID="eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"
Oct 30 20:06:41.496800 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:06:41.496781    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de\": container with ID starting with eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de not found: ID does not exist" containerID="eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"
Oct 30 20:06:41.496867 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.496808    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de"} err="failed to get container status \"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de\": rpc error: code = NotFound desc = could not find container \"eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de\": container with ID starting with eba16fbc9046d5fe710ef3c55144011524265d28ee41b60c3b6633547768a3de not found: ID does not exist"
Oct 30 20:06:41.534836 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.534797    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 20:06:41.546473 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:41.546445    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-54hm4"]
Oct 30 20:06:42.094615 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:42.094566    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" path="/var/lib/kubelet/pods/8eae4b64-b77f-46b0-9e14-19b61a39b7d3/volumes"
Oct 30 20:06:46.661112 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:06:46.661074    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:06:58.645665 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:06:58.645621    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:07:14.798248 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:14.798211    4320 scope.go:122] "RemoveContainer" containerID="c809092193d49a86d4378dd0861824cb6464926303d7d3d3899fd9a6013df5c6"
Oct 30 20:07:20.347841 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:20.347804    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:07:22.885625 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:22.881692    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-d8qbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:22.885625 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:22.881771    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:28.645682 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:07:28.645634    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:07:30.713855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:30.713823    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:07:34.851255 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:34.851220    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:07:49.028179 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:49.028141    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"]
Oct 30 20:07:49.039764 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:49.039736    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c73a9d-hdt97"]
Oct 30 20:07:50.094702 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:50.094660    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80984c1d-f330-4004-8086-d143a6a6e93e" path="/var/lib/kubelet/pods/80984c1d-f330-4004-8086-d143a6a6e93e/volumes"
Oct 30 20:07:53.323209 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:53.323173    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:07:58.045793 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:58.045691    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-j8c67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:58.045793 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:58.045765    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-j8c67" podUID="cede89a0-f33e-4db3-bcac-24ce856b418f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.154454 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:07:58.154416    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:07:58.652468 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:07:58.652392    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:08:14.742513 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:14.742475    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:08:19.350845 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:19.350812    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:08:20.033211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:20.033170    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"]
Oct 30 20:08:20.044432 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:20.044399    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b8daa9-75d7d"]
Oct 30 20:08:20.097009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:20.096974    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b81ca0d-2bd1-4762-85f9-a9288645edbb" path="/var/lib/kubelet/pods/5b81ca0d-2bd1-4762-85f9-a9288645edbb/volumes"
Oct 30 20:08:28.644525 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:08:28.644486    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:08:35.930116 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:35.930079    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:08:40.601539 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:40.601504    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:08:57.021357 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:08:57.021322    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:08:58.644255 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:08:58.644207    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:09:07.822473 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:07.822432    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:09:12.753714 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:12.753679    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:09:14.830205 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:14.830171    4320 scope.go:122] "RemoveContainer" containerID="907949f6d553b64e80a891776307f369ee7e1e4307cb8e16322f5c3e410d4bf5"
Oct 30 20:09:19.029331 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:19.029287    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"]
Oct 30 20:09:19.041142 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:19.041112    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e27f339-rndnd"]
Oct 30 20:09:20.026570 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:20.026534    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"]
Oct 30 20:09:20.038211 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:20.038181    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc11968-lx9zf"]
Oct 30 20:09:20.094672 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:20.094646    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a1352d-4307-4a85-b613-f4e606497350" path="/var/lib/kubelet/pods/06a1352d-4307-4a85-b613-f4e606497350/volumes"
Oct 30 20:09:20.095038 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:20.095024    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6009f59a-c864-4c10-bbeb-81b4963ba505" path="/var/lib/kubelet/pods/6009f59a-c864-4c10-bbeb-81b4963ba505/volumes"
Oct 30 20:09:28.642630 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:09:28.642573    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:09:29.945526 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:29.945488    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:09:34.631859 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:09:34.631823    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:09:58.643262 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:09:58.643224    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:10:03.933725 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:03.933684    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:10:08.649047 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:08.649010    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:10:14.851589 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:14.851552    4320 scope.go:122] "RemoveContainer" containerID="54e96a4c435ddb6687120b88a2f077faf42e6ec8a786794a3c2c13d0499ee007"
Oct 30 20:10:14.859858 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:14.859838    4320 scope.go:122] "RemoveContainer" containerID="1eb4aec036d11f4008fe4d1fe1c760e8004a564d0a99f556c6294ccf2905bd5f"
Oct 30 20:10:14.867519 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:14.867502    4320 scope.go:122] "RemoveContainer" containerID="cfc35a3ac6d323a244f813916d730ca8102cd70750f040e4beac00de59dfa0e8"
Oct 30 20:10:14.874826 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:14.874808    4320 scope.go:122] "RemoveContainer" containerID="ec19d6d0ed979a03d64fa468e1fdab21f51de1936be5bfaa1049ab879deb6a10"
Oct 30 20:10:23.238510 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:23.238474    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:10:28.644017 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:10:28.643975    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:10:35.360612 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.360563    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:10:35.361124 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.360994    4320 cpu_manager.go:414] "containerMap: removing container" podUID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" containerName="discovery"
Oct 30 20:10:35.361124 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.361009    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" containerName="discovery"
Oct 30 20:10:35.361124 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.361109    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eae4b64-b77f-46b0-9e14-19b61a39b7d3" containerName="discovery"
Oct 30 20:10:35.364278 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.364254    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.379796 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.379770    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-013647\"/\"kube-root-ca.crt\""
Oct 30 20:10:35.391858 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.391828    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-013647\"/\"default-dockercfg-7b9nf\""
Oct 30 20:10:35.391858 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.391846    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-013647\"/\"simulator-single-main-013647-kserve-self-signed-certs\""
Oct 30 20:10:35.392027 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.391850    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-013647\"/\"openshift-service-ca.crt\""
Oct 30 20:10:35.393930 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.393907    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:10:35.462317 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462284    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.462487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462323    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.462487 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462464    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbftv\" (UniqueName: \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.462591 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462552    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.462671 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462585    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.462742 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.462725    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563541 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563512    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbftv\" (UniqueName: \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563563    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563580    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563631    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563666    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.563696 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563682    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.564009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.563983    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.564108 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.564086    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.564157 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.564108    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.565854 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.565832    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.566107 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.566090    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.584065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.584042    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbftv\" (UniqueName: \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\") pod \"simulator-single-main-013647-kserve-659f7bf697-dm7wf\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") " pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.685981 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.685934    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:35.987318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:35.987292    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:10:35.990068 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:10:35.990040    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67868cce_7423_4ee8_b181_52790cbcdda6.slice/crio-eaf2bb696a480ffdb4436da9124ac665e7fc7c70601d69d1e6e72f0154f16560 WatchSource:0}: Error finding container eaf2bb696a480ffdb4436da9124ac665e7fc7c70601d69d1e6e72f0154f16560: Status 404 returned error can't find the container with id eaf2bb696a480ffdb4436da9124ac665e7fc7c70601d69d1e6e72f0154f16560
Oct 30 20:10:36.832945 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:36.832914    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" event={"ID":"67868cce-7423-4ee8-b181-52790cbcdda6","Type":"ContainerStarted","Data":"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"}
Oct 30 20:10:36.833330 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:36.832953    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" event={"ID":"67868cce-7423-4ee8-b181-52790cbcdda6","Type":"ContainerStarted","Data":"eaf2bb696a480ffdb4436da9124ac665e7fc7c70601d69d1e6e72f0154f16560"}
Oct 30 20:10:36.872220 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:36.872178    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" podStartSLOduration=1.872164999 podStartE2EDuration="1.872164999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:36.871078952 +0000 UTC m=+3984.906591417" watchObservedRunningTime="2026-10-30 20:10:36.872164999 +0000 UTC m=+3984.907677457"
Oct 30 20:10:37.686888 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:37.686846    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:37.857784 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:37.857753    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:10:45.686829 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:45.686796    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:45.699289 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:45.699267    4320 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:45.884918 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:10:45.884895    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:10:58.643729 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:10:58.643645    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:11:03.129033 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:03.128997    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:11:14.904927 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:14.904895    4320 scope.go:122] "RemoveContainer" containerID="146f4c81aca1012b314ce11647b39f540f07c8243e9a1fecb96307d30f03c176"
Oct 30 20:11:14.913057 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:14.913031    4320 scope.go:122] "RemoveContainer" containerID="6fe77ff6099134667cde5c1ca0c3873699a3af89959af34b52f5c2719037371e"
Oct 30 20:11:20.536164 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:20.536128    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:11:25.163445 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:25.163409    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:11:28.643157 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:11:28.643118    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:11:35.522004 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:35.521963    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:11:40.183366 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.183306    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" podUID="67868cce-7423-4ee8-b181-52790cbcdda6" containerName="main" containerID="cri-o://87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038" gracePeriod=45
Oct 30 20:11:40.423982 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.423955    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:11:40.544090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544012    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544074    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544128    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\" (UniqueName: \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544212    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544275    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\" (UniqueName: \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544272    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:40.544307 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544305    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\") pod \"67868cce-7423-4ee8-b181-52790cbcdda6\" (UID: \"67868cce-7423-4ee8-b181-52790cbcdda6\") "
Oct 30 20:11:40.544584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544340    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:40.544584 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544487    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:40.544772 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544741    4320 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-tmp-dir\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:40.544842 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544775    4320 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-home\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:40.544842 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.544789    4320 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-model-cache\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:40.546511 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.546488    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:40.546667 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.546578    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:40.546730 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.546707    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv" pod "67868cce-7423-4ee8-b181-52790cbcdda6" (UID: "67868cce-7423-4ee8-b181-52790cbcdda6"). InnerVolumeSpecName "kube-api-access-bbftv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:40.645777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.645741    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbftv\" (UniqueName: \"kubernetes.io/projected/67868cce-7423-4ee8-b181-52790cbcdda6-kube-api-access-bbftv\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:40.645777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.645773    4320 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67868cce-7423-4ee8-b181-52790cbcdda6-tls-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:40.645777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:40.645783    4320 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67868cce-7423-4ee8-b181-52790cbcdda6-dshm\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:11:41.137836 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.137797    4320 generic.go:356] "Generic (PLEG): container finished" podID="67868cce-7423-4ee8-b181-52790cbcdda6" containerID="87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038" exitCode=0
Oct 30 20:11:41.138009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.137843    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"
Oct 30 20:11:41.138009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.137894    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" event={"ID":"67868cce-7423-4ee8-b181-52790cbcdda6","Type":"ContainerDied","Data":"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"}
Oct 30 20:11:41.138009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.137929    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf" event={"ID":"67868cce-7423-4ee8-b181-52790cbcdda6","Type":"ContainerDied","Data":"eaf2bb696a480ffdb4436da9124ac665e7fc7c70601d69d1e6e72f0154f16560"}
Oct 30 20:11:41.138009 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.137945    4320 scope.go:122] "RemoveContainer" containerID="87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"
Oct 30 20:11:41.147275 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.147259    4320 scope.go:122] "RemoveContainer" containerID="87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"
Oct 30 20:11:41.147554 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:11:41.147532    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038\": container with ID starting with 87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038 not found: ID does not exist" containerID="87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"
Oct 30 20:11:41.147695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.147560    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038"} err="failed to get container status \"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038\": rpc error: code = NotFound desc = could not find container \"87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038\": container with ID starting with 87c173ad67d235f72dd82e1d02c0014b4d67ba0dfe84801651477c97443c4038 not found: ID does not exist"
Oct 30 20:11:41.186983 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.186961    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:11:41.200480 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:41.200453    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-013647/simulator-single-main-013647-kserve-659f7bf697-dm7wf"]
Oct 30 20:11:42.094950 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:42.094911    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67868cce-7423-4ee8-b181-52790cbcdda6" path="/var/lib/kubelet/pods/67868cce-7423-4ee8-b181-52790cbcdda6/volumes"
Oct 30 20:11:54.169190 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:54.169155    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:11:58.642742 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:11:58.642704    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:11:59.451868 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:11:59.451833    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:12:28.644092 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:12:28.644053    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:12:58.643170 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:12:58.643130    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:13:26.724789 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:13:26.724759    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:13:28.663173 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:13:28.663140    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:13:41.812429 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:13:41.812389    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:13:58.644821 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:13:58.644726    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:14:27.238785 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:14:27.238725    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:14:28.643818 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:14:28.643777    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:14:51.113323 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:14:51.113279    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:14:58.644302 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:14:58.644252    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:15:00.142461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.142430    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:00.142889 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.142874    4320 cpu_manager.go:414] "containerMap: removing container" podUID="67868cce-7423-4ee8-b181-52790cbcdda6" containerName="main"
Oct 30 20:15:00.142939 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.142889    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67868cce-7423-4ee8-b181-52790cbcdda6" containerName="main"
Oct 30 20:15:00.142981 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.142951    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="67868cce-7423-4ee8-b181-52790cbcdda6" containerName="main"
Oct 30 20:15:00.146061 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.146040    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.162130 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.162103    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:15:00.174095 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.174059    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:15:00.175806 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.175780    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:00.259077 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.259045    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:15:00.268505 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.268473    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfx8z\" (UniqueName: \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.268746 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.268721    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.370080 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.370046    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.370303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.370102    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfx8z\" (UniqueName: \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.370820 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.370794    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.389024 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.388991    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfx8z\" (UniqueName: \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\") pod \"maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.459370 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.459289    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:00.755639 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:00.755593    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:00.758716 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:15:00.758686    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30646b9f_d983_472e_a4f9_97d2676893a0.slice/crio-9bd6e0da9c6155c157f56f243f0e23d5f6b8ffbf29b3bca7fa67dd02430e4f64 WatchSource:0}: Error finding container 9bd6e0da9c6155c157f56f243f0e23d5f6b8ffbf29b3bca7fa67dd02430e4f64: Status 404 returned error can't find the container with id 9bd6e0da9c6155c157f56f243f0e23d5f6b8ffbf29b3bca7fa67dd02430e4f64
Oct 30 20:15:01.099472 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:01.099434    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerStarted","Data":"5734ea4ccdf87f08bd407ea24534f12a9a243885155e185078b73f450f6263ec"}
Oct 30 20:15:01.099651 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:01.099492    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerStarted","Data":"9bd6e0da9c6155c157f56f243f0e23d5f6b8ffbf29b3bca7fa67dd02430e4f64"}
Oct 30 20:15:01.138951 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:01.138903    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" podStartSLOduration=1.138877813 podStartE2EDuration="1.138877813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.1385831 +0000 UTC m=+4249.174095564" watchObservedRunningTime="2026-10-30 20:15:01.138877813 +0000 UTC m=+4249.174390276"
Oct 30 20:15:02.105321 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:02.105280    4320 generic.go:356] "Generic (PLEG): container finished" podID="30646b9f-d983-472e-a4f9-97d2676893a0" containerID="5734ea4ccdf87f08bd407ea24534f12a9a243885155e185078b73f450f6263ec" exitCode=60
Oct 30 20:15:02.105860 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:02.105358    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerDied","Data":"5734ea4ccdf87f08bd407ea24534f12a9a243885155e185078b73f450f6263ec"}
Oct 30 20:15:02.105860 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:02.105631    4320 scope.go:122] "RemoveContainer" containerID="5734ea4ccdf87f08bd407ea24534f12a9a243885155e185078b73f450f6263ec"
Oct 30 20:15:03.112247 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:03.112154    4320 generic.go:356] "Generic (PLEG): container finished" podID="30646b9f-d983-472e-a4f9-97d2676893a0" containerID="5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f" exitCode=60
Oct 30 20:15:03.112247 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:03.112218    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerDied","Data":"5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f"}
Oct 30 20:15:03.112768 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:03.112271    4320 scope.go:122] "RemoveContainer" containerID="5734ea4ccdf87f08bd407ea24534f12a9a243885155e185078b73f450f6263ec"
Oct 30 20:15:03.112768 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:03.112551    4320 scope.go:122] "RemoveContainer" containerID="5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f"
Oct 30 20:15:03.112881 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:15:03.112857    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb_odh-ai-gateway-infra(30646b9f-d983-472e-a4f9-97d2676893a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" podUID="30646b9f-d983-472e-a4f9-97d2676893a0"
Oct 30 20:15:04.119075 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:04.119046    4320 scope.go:122] "RemoveContainer" containerID="5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f"
Oct 30 20:15:04.119519 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:15:04.119351    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb_odh-ai-gateway-infra(30646b9f-d983-472e-a4f9-97d2676893a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" podUID="30646b9f-d983-472e-a4f9-97d2676893a0"
Oct 30 20:15:16.754954 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:16.754922    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:15:17.089522 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:17.089441    4320 scope.go:122] "RemoveContainer" containerID="5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f"
Oct 30 20:15:18.185723 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:18.185686    4320 generic.go:356] "Generic (PLEG): container finished" podID="30646b9f-d983-472e-a4f9-97d2676893a0" containerID="c33d37a3dd5f59713ee3b3bfc09ce3343959d5724747690ba6c40f53e5cc3bf7" exitCode=60
Oct 30 20:15:18.186192 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:18.185741    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerDied","Data":"c33d37a3dd5f59713ee3b3bfc09ce3343959d5724747690ba6c40f53e5cc3bf7"}
Oct 30 20:15:18.186192 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:18.185787    4320 scope.go:122] "RemoveContainer" containerID="5ae40cfdc221d1889359736664116746fceecf92566024722c930e25a3faa53f"
Oct 30 20:15:18.186192 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:18.186114    4320 scope.go:122] "RemoveContainer" containerID="c33d37a3dd5f59713ee3b3bfc09ce3343959d5724747690ba6c40f53e5cc3bf7"
Oct 30 20:15:18.186413 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:15:18.186388    4320 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb_odh-ai-gateway-infra(30646b9f-d983-472e-a4f9-97d2676893a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" podUID="30646b9f-d983-472e-a4f9-97d2676893a0"
Oct 30 20:15:19.232126 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.232088    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:19.374115 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.374087    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:19.459299 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.459258    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\") pod \"30646b9f-d983-472e-a4f9-97d2676893a0\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") "
Oct 30 20:15:19.459521 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.459360    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\" (UniqueName: \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\") pod \"30646b9f-d983-472e-a4f9-97d2676893a0\" (UID: \"30646b9f-d983-472e-a4f9-97d2676893a0\") "
Oct 30 20:15:19.459719 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.459691    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca" pod "30646b9f-d983-472e-a4f9-97d2676893a0" (UID: "30646b9f-d983-472e-a4f9-97d2676893a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.459860 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.459843    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30646b9f-d983-472e-a4f9-97d2676893a0-maas-api-service-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:15:19.461455 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.461435    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z" pod "30646b9f-d983-472e-a4f9-97d2676893a0" (UID: "30646b9f-d983-472e-a4f9-97d2676893a0"). InnerVolumeSpecName "kube-api-access-nfx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.560592 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:19.560486    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfx8z\" (UniqueName: \"kubernetes.io/projected/30646b9f-d983-472e-a4f9-97d2676893a0-kube-api-access-nfx8z\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:15:20.197792 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:20.197758    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb" event={"ID":"30646b9f-d983-472e-a4f9-97d2676893a0","Type":"ContainerDied","Data":"9bd6e0da9c6155c157f56f243f0e23d5f6b8ffbf29b3bca7fa67dd02430e4f64"}
Oct 30 20:15:20.197792 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:20.197779    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"
Oct 30 20:15:20.198052 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:20.197820    4320 scope.go:122] "RemoveContainer" containerID="c33d37a3dd5f59713ee3b3bfc09ce3343959d5724747690ba6c40f53e5cc3bf7"
Oct 30 20:15:20.246514 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:20.246485    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:20.258198 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:20.258170    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-0bdca7-2988985bcncb"]
Oct 30 20:15:22.094503 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:22.094466    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" path="/var/lib/kubelet/pods/30646b9f-d983-472e-a4f9-97d2676893a0/volumes"
Oct 30 20:15:28.644244 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:15:28.644207    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:15:40.230040 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:40.229996    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:15:54.835780 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:15:54.835735    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:15:58.644521 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:15:58.644479    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:16:28.646480 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:16:28.646439    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:16:30.830026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:16:30.829987    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:16:58.644899 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:16:58.644816    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:16:59.615283 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:16:59.615242    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:17:28.643884 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:17:28.643843    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:17:35.624500 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:17:35.624458    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:17:38.440801 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:17:38.440749    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:17:38.441536 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:17:38.441499    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:17:46.721778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:17:46.721737    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:17:57.841079 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:17:57.841034    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:17:58.648474 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:17:58.648432    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:18:02.943386 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:02.943345    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:18:08.440626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:08.439731    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:08.440626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:08.439826    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:08.622217 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:08.622179    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:18:16.751409 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:16.751365    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:18:20.711020 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:20.710980    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:18:23.443361 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:23.443305    4320 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-nv972 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:23.444573 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:23.444094    4320 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:28.644839 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:18:28.644801    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:18:37.056101 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:18:37.056052    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:18:58.644318 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:18:58.644281    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:19:17.010788 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:17.010752    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:19:22.220485 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:22.220451    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:19:26.288658 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.288623    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"]
Oct 30 20:19:26.289065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289023    4320 cpu_manager.go:414] "containerMap: removing container" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289035    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289048    4320 cpu_manager.go:414] "containerMap: removing container" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289053    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289065 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289063    4320 cpu_manager.go:414] "containerMap: removing container" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289265 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289068    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289265 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289136    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289265 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289144    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.289265 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.289152    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="30646b9f-d983-472e-a4f9-97d2676893a0" containerName="cleanup"
Oct 30 20:19:26.292193 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.292177    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.310854 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.310830    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"]
Oct 30 20:19:26.320698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.320671    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:26.320823 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.320743    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:19:26.351422 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.351395    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.351538 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.351440    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4sw7\" (UniqueName: \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.351538 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.351462    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.376845 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.376817    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:26.376951 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.376864    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:26.452771 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.452742    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.452889 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.452793    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4sw7\" (UniqueName: \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.452889 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.452815    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.453338 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.453313    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.455060 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.455035    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.471575 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.471552    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4sw7\" (UniqueName: \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\") pod \"maas-api-revoke-sub-f03629db-sf6l8\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.604890 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.604837    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:26.897469 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:26.897438    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"]
Oct 30 20:19:26.900665 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:19:26.900634    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e6d281_6891_4b13_a910_4121df76fbac.slice/crio-f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f WatchSource:0}: Error finding container f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f: Status 404 returned error can't find the container with id f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f
Oct 30 20:19:27.468283 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:27.468248    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8" event={"ID":"e0e6d281-6891-4b13-a910-4121df76fbac","Type":"ContainerStarted","Data":"f55cb525122531501f60fc24cfc5c93bc51d6d12da2489b85bfd435b3d9bb034"}
Oct 30 20:19:27.468645 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:27.468306    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8" event={"ID":"e0e6d281-6891-4b13-a910-4121df76fbac","Type":"ContainerStarted","Data":"f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f"}
Oct 30 20:19:27.513950 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:27.513906    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8" podStartSLOduration=1.513893211 podStartE2EDuration="1.513893211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:27.505805794 +0000 UTC m=+4515.541318258" watchObservedRunningTime="2026-10-30 20:19:27.513893211 +0000 UTC m=+4515.549405674"
Oct 30 20:19:28.473933 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:28.473895    4320 generic.go:356] "Generic (PLEG): container finished" podID="e0e6d281-6891-4b13-a910-4121df76fbac" containerID="f55cb525122531501f60fc24cfc5c93bc51d6d12da2489b85bfd435b3d9bb034" exitCode=0
Oct 30 20:19:28.474323 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:28.473989    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8" event={"ID":"e0e6d281-6891-4b13-a910-4121df76fbac","Type":"ContainerDied","Data":"f55cb525122531501f60fc24cfc5c93bc51d6d12da2489b85bfd435b3d9bb034"}
Oct 30 20:19:28.643234 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:19:28.643200    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:19:29.619404 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.619382    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:29.681262 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.681229    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\") pod \"e0e6d281-6891-4b13-a910-4121df76fbac\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") "
Oct 30 20:19:29.681417 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.681273    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\") pod \"e0e6d281-6891-4b13-a910-4121df76fbac\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") "
Oct 30 20:19:29.681417 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.681347    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\" (UniqueName: \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\") pod \"e0e6d281-6891-4b13-a910-4121df76fbac\" (UID: \"e0e6d281-6891-4b13-a910-4121df76fbac\") "
Oct 30 20:19:29.681717 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.681617    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca" pod "e0e6d281-6891-4b13-a910-4121df76fbac" (UID: "e0e6d281-6891-4b13-a910-4121df76fbac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:29.681833 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.681732    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:29.683995 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.683965    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca" pod "e0e6d281-6891-4b13-a910-4121df76fbac" (UID: "e0e6d281-6891-4b13-a910-4121df76fbac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:29.684109 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.684036    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7" pod "e0e6d281-6891-4b13-a910-4121df76fbac" (UID: "e0e6d281-6891-4b13-a910-4121df76fbac"). InnerVolumeSpecName "kube-api-access-q4sw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:29.782793 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.782742    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4sw7\" (UniqueName: \"kubernetes.io/projected/e0e6d281-6891-4b13-a910-4121df76fbac-kube-api-access-q4sw7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:29.782793 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:29.782771    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e6d281-6891-4b13-a910-4121df76fbac-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:30.484777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:30.484751    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"
Oct 30 20:19:30.484777 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:30.484765    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8" event={"ID":"e0e6d281-6891-4b13-a910-4121df76fbac","Type":"ContainerDied","Data":"f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f"}
Oct 30 20:19:30.484988 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:30.484810    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f79357c8df7c460cf547002f309f8596bfe812eb27f6702b02155f328068993f"
Oct 30 20:19:37.428611 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:37.428558    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:19:42.874090 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.874055    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"]
Oct 30 20:19:42.874698 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.874676    4320 cpu_manager.go:414] "containerMap: removing container" podUID="e0e6d281-6891-4b13-a910-4121df76fbac" containerName="revoke-keys"
Oct 30 20:19:42.874807 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.874700    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e6d281-6891-4b13-a910-4121df76fbac" containerName="revoke-keys"
Oct 30 20:19:42.874874 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.874839    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e6d281-6891-4b13-a910-4121df76fbac" containerName="revoke-keys"
Oct 30 20:19:42.878185 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.878163    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:42.895565 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.895530    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"]
Oct 30 20:19:42.905471 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.905440    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:42.905666 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.905517    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:42.905666 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.905582    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:19:42.905666 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:42.905446    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:43.044468 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.044428    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.044694 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.044571    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.044694 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.044630    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wvs\" (UniqueName: \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.145666 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.145636    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.145840 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.145679    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wvs\" (UniqueName: \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.145840 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.145728    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.146272 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.146244    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.148025 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.148005    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.164212 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.164189    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wvs\" (UniqueName: \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\") pod \"maas-api-revoke-sub-0165d7d2-zr4vz\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.191099 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.191078    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:43.485475 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.485445    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"]
Oct 30 20:19:43.488393 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:19:43.488345    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1ceed7_d753_42f5_99cc_660d37fde5f3.slice/crio-ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18 WatchSource:0}: Error finding container ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18: Status 404 returned error can't find the container with id ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18
Oct 30 20:19:43.551739 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:43.551709    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz" event={"ID":"1a1ceed7-d753-42f5-99cc-660d37fde5f3","Type":"ContainerStarted","Data":"ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18"}
Oct 30 20:19:44.557976 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:44.557902    4320 generic.go:356] "Generic (PLEG): container finished" podID="1a1ceed7-d753-42f5-99cc-660d37fde5f3" containerID="974242dfa629e03148bc7513f77b1975d753bf9b70c937bcace9f680af7aec40" exitCode=0
Oct 30 20:19:44.557976 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:44.557967    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz" event={"ID":"1a1ceed7-d753-42f5-99cc-660d37fde5f3","Type":"ContainerDied","Data":"974242dfa629e03148bc7513f77b1975d753bf9b70c937bcace9f680af7aec40"}
Oct 30 20:19:45.702484 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.702461    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:45.870909 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.870837    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\") pod \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") "
Oct 30 20:19:45.870909 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.870883    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\") pod \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") "
Oct 30 20:19:45.871099 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.870941    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\" (UniqueName: \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\") pod \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\" (UID: \"1a1ceed7-d753-42f5-99cc-660d37fde5f3\") "
Oct 30 20:19:45.871200 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.871178    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca" pod "1a1ceed7-d753-42f5-99cc-660d37fde5f3" (UID: "1a1ceed7-d753-42f5-99cc-660d37fde5f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:45.873155 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.873130    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca" pod "1a1ceed7-d753-42f5-99cc-660d37fde5f3" (UID: "1a1ceed7-d753-42f5-99cc-660d37fde5f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:45.873275 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.873165    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs" pod "1a1ceed7-d753-42f5-99cc-660d37fde5f3" (UID: "1a1ceed7-d753-42f5-99cc-660d37fde5f3"). InnerVolumeSpecName "kube-api-access-t9wvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:45.971917 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.971893    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:45.971917 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.971914    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1ceed7-d753-42f5-99cc-660d37fde5f3-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:45.972066 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:45.971924    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wvs\" (UniqueName: \"kubernetes.io/projected/1a1ceed7-d753-42f5-99cc-660d37fde5f3-kube-api-access-t9wvs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:46.568823 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:46.568793    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"
Oct 30 20:19:46.568983 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:46.568819    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz" event={"ID":"1a1ceed7-d753-42f5-99cc-660d37fde5f3","Type":"ContainerDied","Data":"ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18"}
Oct 30 20:19:46.568983 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:46.568869    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae2ed0481449447d3c1b1de2b8b326bb160512e9f9bf55cfcc252ddaa8f68f18"
Oct 30 20:19:53.787653 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.787617    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"]
Oct 30 20:19:53.788093 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.788020    4320 cpu_manager.go:414] "containerMap: removing container" podUID="1a1ceed7-d753-42f5-99cc-660d37fde5f3" containerName="revoke-keys"
Oct 30 20:19:53.788093 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.788031    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1ceed7-d753-42f5-99cc-660d37fde5f3" containerName="revoke-keys"
Oct 30 20:19:53.788204 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.788119    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1ceed7-d753-42f5-99cc-660d37fde5f3" containerName="revoke-keys"
Oct 30 20:19:53.791559 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.791536    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.819489 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.819452    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:53.819778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.819451    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:53.819778 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.819509    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\""
Oct 30 20:19:53.819971 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.819523    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:53.821442 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.821419    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"]
Oct 30 20:19:53.836402 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.836362    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxtt\" (UniqueName: \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.836607 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.836545    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.836814 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.836794    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.937912 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.937872    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.938116 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.937946    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.938116 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.937992    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxtt\" (UniqueName: \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.938650 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.938585    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.940339 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.940316    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:53.960012 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:53.959973    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxtt\" (UniqueName: \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\") pod \"maas-api-revoke-sub-6b19338f-bgvl8\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:54.104830 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:54.104736    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:54.410524 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:54.410490    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"]
Oct 30 20:19:54.414255 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:19:54.414225    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cec1b1_9806_4958_82a3_3ed23746d5a9.slice/crio-c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955 WatchSource:0}: Error finding container c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955: Status 404 returned error can't find the container with id c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955
Oct 30 20:19:54.607165 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:54.607134    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8" event={"ID":"98cec1b1-9806-4958-82a3-3ed23746d5a9","Type":"ContainerStarted","Data":"c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955"}
Oct 30 20:19:55.613138 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:55.613045    4320 generic.go:356] "Generic (PLEG): container finished" podID="98cec1b1-9806-4958-82a3-3ed23746d5a9" containerID="7bc2cf330d2a2ffcdbdd513777ef48190267a5e9de2176abf533069356f0d801" exitCode=0
Oct 30 20:19:55.613138 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:55.613129    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8" event={"ID":"98cec1b1-9806-4958-82a3-3ed23746d5a9","Type":"ContainerDied","Data":"7bc2cf330d2a2ffcdbdd513777ef48190267a5e9de2176abf533069356f0d801"}
Oct 30 20:19:56.759795 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.759765    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:56.869579 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.869491    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\") pod \"98cec1b1-9806-4958-82a3-3ed23746d5a9\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") "
Oct 30 20:19:56.869749 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.869583    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\" (UniqueName: \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\") pod \"98cec1b1-9806-4958-82a3-3ed23746d5a9\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") "
Oct 30 20:19:56.869749 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.869706    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\") pod \"98cec1b1-9806-4958-82a3-3ed23746d5a9\" (UID: \"98cec1b1-9806-4958-82a3-3ed23746d5a9\") "
Oct 30 20:19:56.870085 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.870062    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca" pod "98cec1b1-9806-4958-82a3-3ed23746d5a9" (UID: "98cec1b1-9806-4958-82a3-3ed23746d5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:56.870170 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.870147    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-openshift-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:56.871936 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.871906    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca" pod "98cec1b1-9806-4958-82a3-3ed23746d5a9" (UID: "98cec1b1-9806-4958-82a3-3ed23746d5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:56.872018 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.871911    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt" pod "98cec1b1-9806-4958-82a3-3ed23746d5a9" (UID: "98cec1b1-9806-4958-82a3-3ed23746d5a9"). InnerVolumeSpecName "kube-api-access-chxtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:56.970695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.970657    4320 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cec1b1-9806-4958-82a3-3ed23746d5a9-maas-api-xks-ca\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:56.970695 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:56.970688    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxtt\" (UniqueName: \"kubernetes.io/projected/98cec1b1-9806-4958-82a3-3ed23746d5a9-kube-api-access-chxtt\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:19:57.623264 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:57.623230    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"
Oct 30 20:19:57.623264 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:57.623247    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8" event={"ID":"98cec1b1-9806-4958-82a3-3ed23746d5a9","Type":"ContainerDied","Data":"c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955"}
Oct 30 20:19:57.623506 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:19:57.623294    4320 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f2ed8b540ce53c8cd12f0c11b2164ac906b34f08976c1e20e7f0b9cc22c955"
Oct 30 20:19:58.644014 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:19:58.643977    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:20:04.917656 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:04.917612    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mvtfx"]
Oct 30 20:20:28.644394 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:20:28.644355    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:20:36.166626 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:36.166571    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lhwvl_3ec07eb2-2af5-4d2b-ae2d-d905c57085f8/manager/2.log"
Oct 30 20:20:36.392051 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:36.392020    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-7dx5m_61f6a874-9cc9-4ceb-a304-dab54cf6a8be/manager/0.log"
Oct 30 20:20:36.630093 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:36.630014    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-c757k_6ab50bb9-d775-4460-bcff-e18d445dc2a3/manager/0.log"
Oct 30 20:20:37.207559 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:37.207528    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-m6f5w_92948160-ac67-4192-ad60-83336fc1158f/server/0.log"
Oct 30 20:20:37.407784 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:37.407749    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gqrwf_41afb888-9cc2-486a-b5f7-5e6fa8863e06/manager/0.log"
Oct 30 20:20:37.856935 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:37.856904    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/copy-manifests/0.log"
Oct 30 20:20:37.884888 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:37.884858    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/manager/0.log"
Oct 30 20:20:39.070105 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:39.070065    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lhwvl_3ec07eb2-2af5-4d2b-ae2d-d905c57085f8/manager/1.log"
Oct 30 20:20:40.029510 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:40.029479    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zr4vz_1a1ceed7-d753-42f5-99cc-660d37fde5f3/revoke-keys/0.log"
Oct 30 20:20:40.604413 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:40.604385    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bgvl8_98cec1b1-9806-4958-82a3-3ed23746d5a9/revoke-keys/0.log"
Oct 30 20:20:40.794677 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:40.794644    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sf6l8_e0e6d281-6891-4b13-a910-4121df76fbac/revoke-keys/0.log"
Oct 30 20:20:43.818104 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:43.818068    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mvtfx_6faa0024-5182-47d7-96d2-043069294f99/limitador/0.log"
Oct 30 20:20:44.797813 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:44.797780    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c574v98c_7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca/istio-proxy/0.log"
Oct 30 20:20:46.208933 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:46.208902    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-d8qbx_df9d0999-76b8-45bd-87db-fd086b0ad274/istio-proxy/0.log"
Oct 30 20:20:46.624446 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:46.624360    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-j8c67_cede89a0-f33e-4db3-bcac-24ce856b418f/istio-proxy/0.log"
Oct 30 20:20:47.041439 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:20:47.041407    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-nv972_fcd4fcad-85bf-4edb-b322-fbc4defad9ce/istio-proxy/0.log"
Oct 30 20:20:58.643322 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:20:58.643276    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:21:28.644124 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:21:28.644029    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:21:30.654257 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:30.654226    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lhwvl_3ec07eb2-2af5-4d2b-ae2d-d905c57085f8/manager/2.log"
Oct 30 20:21:30.897951 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:30.897911    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-7dx5m_61f6a874-9cc9-4ceb-a304-dab54cf6a8be/manager/0.log"
Oct 30 20:21:31.133668 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:31.133644    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-c757k_6ab50bb9-d775-4460-bcff-e18d445dc2a3/manager/0.log"
Oct 30 20:21:41.617786 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:41.617755    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-m6f5w_92948160-ac67-4192-ad60-83336fc1158f/server/0.log"
Oct 30 20:21:41.829360 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:41.829319    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gqrwf_41afb888-9cc2-486a-b5f7-5e6fa8863e06/manager/0.log"
Oct 30 20:21:42.355665 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:42.355632    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/copy-manifests/0.log"
Oct 30 20:21:42.386781 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:42.386746    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/manager/0.log"
Oct 30 20:21:54.112356 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:54.112317    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zr4vz_1a1ceed7-d753-42f5-99cc-660d37fde5f3/revoke-keys/0.log"
Oct 30 20:21:54.708136 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:54.708103    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bgvl8_98cec1b1-9806-4958-82a3-3ed23746d5a9/revoke-keys/0.log"
Oct 30 20:21:54.901862 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:54.901828    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sf6l8_e0e6d281-6891-4b13-a910-4121df76fbac/revoke-keys/0.log"
Oct 30 20:21:57.574218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:57.574186    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mvtfx_6faa0024-5182-47d7-96d2-043069294f99/limitador/0.log"
Oct 30 20:21:58.383503 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:58.383464    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c574v98c_7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca/istio-proxy/0.log"
Oct 30 20:21:58.644022 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:21:58.643933    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:21:59.802015 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:21:59.801977    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-d8qbx_df9d0999-76b8-45bd-87db-fd086b0ad274/istio-proxy/0.log"
Oct 30 20:22:00.226100 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:22:00.226072    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-j8c67_cede89a0-f33e-4db3-bcac-24ce856b418f/istio-proxy/0.log"
Oct 30 20:22:00.632510 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:22:00.632425    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-nv972_fcd4fcad-85bf-4edb-b322-fbc4defad9ce/istio-proxy/0.log"
Oct 30 20:22:28.643046 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:22:28.643001    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:22:58.643345 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:22:58.643265    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:23:28.644072 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:23:28.644025    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:23:58.642682 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:23:58.642636    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:24:28.643838 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:24:28.643755    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:24:30.027386 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:30.027351    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"]
Oct 30 20:24:30.038860 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:30.038831    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sf6l8"]
Oct 30 20:24:30.094469 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:30.094439    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e6d281-6891-4b13-a910-4121df76fbac" path="/var/lib/kubelet/pods/e0e6d281-6891-4b13-a910-4121df76fbac/volumes"
Oct 30 20:24:46.026143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:46.026101    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"]
Oct 30 20:24:46.037839 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:46.037809    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zr4vz"]
Oct 30 20:24:46.094635 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:46.094608    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1ceed7-d753-42f5-99cc-660d37fde5f3" path="/var/lib/kubelet/pods/1a1ceed7-d753-42f5-99cc-660d37fde5f3/volumes"
Oct 30 20:24:57.026578 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:57.026539    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"]
Oct 30 20:24:57.038183 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:57.038143    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bgvl8"]
Oct 30 20:24:58.094614 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:24:58.094568    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cec1b1-9806-4958-82a3-3ed23746d5a9" path="/var/lib/kubelet/pods/98cec1b1-9806-4958-82a3-3ed23746d5a9/volumes"
Oct 30 20:24:58.646448 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:24:58.646408    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:25:05.813440 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.813401    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjhm6/must-gather-cth5j"]
Oct 30 20:25:05.814064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.813994    4320 cpu_manager.go:414] "containerMap: removing container" podUID="98cec1b1-9806-4958-82a3-3ed23746d5a9" containerName="revoke-keys"
Oct 30 20:25:05.814064 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.814015    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cec1b1-9806-4958-82a3-3ed23746d5a9" containerName="revoke-keys"
Oct 30 20:25:05.814206 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.814156    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cec1b1-9806-4958-82a3-3ed23746d5a9" containerName="revoke-keys"
Oct 30 20:25:05.817433 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.817410    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:05.843807 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.843783    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"openshift-service-ca.crt\""
Oct 30 20:25:05.843957 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.843795    4320 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjhm6\"/\"default-dockercfg-st55s\""
Oct 30 20:25:05.843957 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.843818    4320 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"kube-root-ca.crt\""
Oct 30 20:25:05.846377 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.846350    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/must-gather-cth5j"]
Oct 30 20:25:05.924334 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.924312    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9578cfd-e090-4d21-9de6-dca965eccdf6-must-gather-output\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:05.924452 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:05.924389    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhfj\" (UniqueName: \"kubernetes.io/projected/c9578cfd-e090-4d21-9de6-dca965eccdf6-kube-api-access-llhfj\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.025806 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.025777    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9578cfd-e090-4d21-9de6-dca965eccdf6-must-gather-output\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.025965 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.025850    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhfj\" (UniqueName: \"kubernetes.io/projected/c9578cfd-e090-4d21-9de6-dca965eccdf6-kube-api-access-llhfj\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.026146 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.026104    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9578cfd-e090-4d21-9de6-dca965eccdf6-must-gather-output\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.044823 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.044798    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhfj\" (UniqueName: \"kubernetes.io/projected/c9578cfd-e090-4d21-9de6-dca965eccdf6-kube-api-access-llhfj\") pod \"must-gather-cth5j\" (UID: \"c9578cfd-e090-4d21-9de6-dca965eccdf6\") " pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.144074 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.144053    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/must-gather-cth5j"
Oct 30 20:25:06.438944 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.438745    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/must-gather-cth5j"]
Oct 30 20:25:06.441881 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:25:06.441851    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9578cfd_e090_4d21_9de6_dca965eccdf6.slice/crio-300cbed31d77ce1c4fb771093b7e6350fb7ba662c7a68a7ba9034dfc95a27202 WatchSource:0}: Error finding container 300cbed31d77ce1c4fb771093b7e6350fb7ba662c7a68a7ba9034dfc95a27202: Status 404 returned error can't find the container with id 300cbed31d77ce1c4fb771093b7e6350fb7ba662c7a68a7ba9034dfc95a27202
Oct 30 20:25:06.443741 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:06.443723    4320 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:25:07.062025 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:07.061990    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/must-gather-cth5j" event={"ID":"c9578cfd-e090-4d21-9de6-dca965eccdf6","Type":"ContainerStarted","Data":"300cbed31d77ce1c4fb771093b7e6350fb7ba662c7a68a7ba9034dfc95a27202"}
Oct 30 20:25:08.068155 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:08.068123    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/must-gather-cth5j" event={"ID":"c9578cfd-e090-4d21-9de6-dca965eccdf6","Type":"ContainerStarted","Data":"4db0b8f76d0037087c26e68c1f76d61bd280678906e424f6e579fed4c833bcc8"}
Oct 30 20:25:08.068593 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:08.068166    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/must-gather-cth5j" event={"ID":"c9578cfd-e090-4d21-9de6-dca965eccdf6","Type":"ContainerStarted","Data":"1bc44b809351b945bd0dc1850c1f0e143f48e50a0bcbf3e1c3b041622e844f0b"}
Oct 30 20:25:08.108172 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:08.108101    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjhm6/must-gather-cth5j" podStartSLOduration=2.452469305 podStartE2EDuration="3.108081497s" totalImagesPullingTime="655.612192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:08.10784401 +0000 UTC m=+4856.143356474" watchObservedRunningTime="2026-10-30 20:25:08.108081497 +0000 UTC m=+4856.143593961"
Oct 30 20:25:10.350824 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:10.350792    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kbslw_0261b7c5-c9bb-4fb6-aa6d-23914249b848/global-pull-secret-syncer/0.log"
Oct 30 20:25:11.200538 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:11.200507    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rrwhk_cbc31816-a783-44bd-83e4-a009e13da8b4/konnectivity-agent/0.log"
Oct 30 20:25:11.300530 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:11.300502    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-217.ec2.internal_32feda196b1df490b5d951791a17beab/haproxy/0.log"
Oct 30 20:25:28.643984 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:25:28.643950    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:25:28.871738 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:28.871702    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:25:28.872080 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:28.871988    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy" containerID="cri-o://a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8" gracePeriod=30
Oct 30 20:25:28.883312 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:28.883286    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:25:28.883477 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:28.883457    4320 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy" containerID="cri-o://c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510" gracePeriod=30
Oct 30 20:25:34.134543 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.134520    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:25:34.166313 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.166293    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:25:34.190310 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.190283    4320 generic.go:356] "Generic (PLEG): container finished" podID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerID="c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510" exitCode=0
Oct 30 20:25:34.190453 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.190329    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"
Oct 30 20:25:34.190511 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.190448    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" event={"ID":"df9d0999-76b8-45bd-87db-fd086b0ad274","Type":"ContainerDied","Data":"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"}
Oct 30 20:25:34.190557 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.190519    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx" event={"ID":"df9d0999-76b8-45bd-87db-fd086b0ad274","Type":"ContainerDied","Data":"f54630ebc7137f29cf12640513cb8157d18cd523f2a8e1276e599a6661e35831"}
Oct 30 20:25:34.190628 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.190552    4320 scope.go:122] "RemoveContainer" containerID="c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"
Oct 30 20:25:34.191894 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.191869    4320 generic.go:356] "Generic (PLEG): container finished" podID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerID="a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8" exitCode=0
Oct 30 20:25:34.192028 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.191924    4320 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"
Oct 30 20:25:34.192028 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.191966    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" event={"ID":"fcd4fcad-85bf-4edb-b322-fbc4defad9ce","Type":"ContainerDied","Data":"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"}
Oct 30 20:25:34.192028 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.192009    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972" event={"ID":"fcd4fcad-85bf-4edb-b322-fbc4defad9ce","Type":"ContainerDied","Data":"0b3cb012be61d05980b9ff6eb53875e4a797977f9f3459b5ba86ce514ec5bfa8"}
Oct 30 20:25:34.201273 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.201248    4320 scope.go:122] "RemoveContainer" containerID="c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"
Oct 30 20:25:34.201742 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:25:34.201589    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510\": container with ID starting with c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510 not found: ID does not exist" containerID="c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"
Oct 30 20:25:34.201742 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.201642    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510"} err="failed to get container status \"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510\": rpc error: code = NotFound desc = could not find container \"c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510\": container with ID starting with c0495e2baf7f785e7ea1f2c1a0f19404bde3ec1776ed2c1f7efb66c90d83c510 not found: ID does not exist"
Oct 30 20:25:34.201742 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.201664    4320 scope.go:122] "RemoveContainer" containerID="a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"
Oct 30 20:25:34.210329 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.210310    4320 scope.go:122] "RemoveContainer" containerID="a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"
Oct 30 20:25:34.210636 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:25:34.210589    4320 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8\": container with ID starting with a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8 not found: ID does not exist" containerID="a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"
Oct 30 20:25:34.210721 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.210646    4320 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8"} err="failed to get container status \"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8\": rpc error: code = NotFound desc = could not find container \"a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8\": container with ID starting with a9b4e0dfdc827bc89835192013c6634a9ec537c51c85ddef2f41b91fc2e2f9a8 not found: ID does not exist"
Oct 30 20:25:34.273121 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273091    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273137    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273175    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273199    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273216    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273263 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273239    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273276    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273297    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273330    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273348    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273383    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273423    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273468    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273508    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273538    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.273560 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273561    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.274088 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273656    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\") pod \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\" (UID: \"fcd4fcad-85bf-4edb-b322-fbc4defad9ce\") "
Oct 30 20:25:34.274088 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.273688    4320 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\") pod \"df9d0999-76b8-45bd-87db-fd086b0ad274\" (UID: \"df9d0999-76b8-45bd-87db-fd086b0ad274\") "
Oct 30 20:25:34.274759 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.274526    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.274759 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.274572    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.274947 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.274822    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.275287 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.275088    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.275662 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.275422    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.275763 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.275737    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:34.276208 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.275822    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:34.276208 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.275969    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.276630 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.276391    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.276955 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.276923    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.277831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.277780    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:34.277941 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.277872    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.278459 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.278435    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "kube-api-access-b899r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.279109 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.279084    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "kube-api-access-9czh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.279267 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.279245    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:34.279509 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.279488    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:34.279887 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.279866    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy" pod "fcd4fcad-85bf-4edb-b322-fbc4defad9ce" (UID: "fcd4fcad-85bf-4edb-b322-fbc4defad9ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.280335 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.280317    4320 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy" pod "df9d0999-76b8-45bd-87db-fd086b0ad274" (UID: "df9d0999-76b8-45bd-87db-fd086b0ad274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:34.374894 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374868    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.374894 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374892    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9d0999-76b8-45bd-87db-fd086b0ad274-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374900    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b899r\" (UniqueName: \"kubernetes.io/projected/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-kube-api-access-b899r\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374910    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374918    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374925    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374932    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374941    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374949    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374957    4320 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-data\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374964    4320 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-podinfo\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374973    4320 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9czh7\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-kube-api-access-9czh7\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374981    4320 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-workload-certs\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374989    4320 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-workload-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.374997    4320 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9d0999-76b8-45bd-87db-fd086b0ad274-credential-socket\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.375005    4320 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istiod-ca-cert\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.375012    4320 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd4fcad-85bf-4edb-b322-fbc4defad9ce-istio-envoy\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.375043 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.375019    4320 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9d0999-76b8-45bd-87db-fd086b0ad274-istio-token\") on node \"ip-10-0-130-217.ec2.internal\" DevicePath \"\""
Oct 30 20:25:34.540175 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.540148    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:25:34.543318 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.543295    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-d8qbx"]
Oct 30 20:25:34.575852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.575833    4320 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:25:34.579570 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:34.579549    4320 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-nv972"]
Oct 30 20:25:36.094274 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:36.094244    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" path="/var/lib/kubelet/pods/df9d0999-76b8-45bd-87db-fd086b0ad274/volumes"
Oct 30 20:25:36.094685 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:25:36.094669    4320 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" path="/var/lib/kubelet/pods/fcd4fcad-85bf-4edb-b322-fbc4defad9ce/volumes"
Oct 30 20:25:58.643709 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:25:58.643627    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:26:07.834534 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:07.834501    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mvtfx_6faa0024-5182-47d7-96d2-043069294f99/limitador/0.log"
Oct 30 20:26:10.246933 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.246897    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v6nx4_3f04f850-94c5-409b-b58a-856349d6683c/kube-state-metrics/0.log"
Oct 30 20:26:10.364963 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.364933    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v6nx4_3f04f850-94c5-409b-b58a-856349d6683c/kube-rbac-proxy-main/0.log"
Oct 30 20:26:10.487341 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.487308    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v6nx4_3f04f850-94c5-409b-b58a-856349d6683c/kube-rbac-proxy-self/0.log"
Oct 30 20:26:10.615405 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.615336    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57c6bff-q9z2d_03034025-6350-4677-976e-17f832f5ee08/metrics-server/0.log"
Oct 30 20:26:10.744973 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.744947    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hvf4z_e86a99a6-dcd2-429d-b35f-3da6b10661ba/monitoring-plugin/0.log"
Oct 30 20:26:10.874063 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.873995    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6xt_42185bd3-6753-4e3b-a38e-5b172172e005/node-exporter/0.log"
Oct 30 20:26:10.997497 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:10.997472    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6xt_42185bd3-6753-4e3b-a38e-5b172172e005/kube-rbac-proxy/0.log"
Oct 30 20:26:11.121150 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:11.121128    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hg6xt_42185bd3-6753-4e3b-a38e-5b172172e005/init-textfile/0.log"
Oct 30 20:26:12.030618 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.030578    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-58tfl_78011371-2b97-4cbd-b99b-bc29a7138874/kube-rbac-proxy-main/0.log"
Oct 30 20:26:12.153323 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.153297    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-58tfl_78011371-2b97-4cbd-b99b-bc29a7138874/kube-rbac-proxy-self/0.log"
Oct 30 20:26:12.277313 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.277282    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-58tfl_78011371-2b97-4cbd-b99b-bc29a7138874/openshift-state-metrics/0.log"
Oct 30 20:26:12.412818 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.412754    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/prometheus/0.log"
Oct 30 20:26:12.534640 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.534586    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/config-reloader/0.log"
Oct 30 20:26:12.660000 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.659975    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/thanos-sidecar/0.log"
Oct 30 20:26:12.782276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.782217    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/kube-rbac-proxy-web/0.log"
Oct 30 20:26:12.907452 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:12.907422    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/kube-rbac-proxy/0.log"
Oct 30 20:26:13.031759 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:13.031727    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/kube-rbac-proxy-thanos/0.log"
Oct 30 20:26:13.153855 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:13.153825    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8e436948-1c32-4bd5-bb49-91f517366e1c/init-config-reloader/0.log"
Oct 30 20:26:13.288841 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:13.288811    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qdnjd_d8b4f72b-b606-4aad-8f47-fc6ef7d146a5/prometheus-operator/0.log"
Oct 30 20:26:13.407442 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:13.407368    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qdnjd_d8b4f72b-b606-4aad-8f47-fc6ef7d146a5/kube-rbac-proxy/0.log"
Oct 30 20:26:13.532185 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:13.532158    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-56cvd_be020691-8510-4e02-9ee5-d01d497d6710/prometheus-operator-admission-webhook/0.log"
Oct 30 20:26:14.027389 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.027364    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/thanos-query/0.log"
Oct 30 20:26:14.148653 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.148623    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/kube-rbac-proxy-web/0.log"
Oct 30 20:26:14.271727 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.271699    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/kube-rbac-proxy/0.log"
Oct 30 20:26:14.394681 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.394656    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/prom-label-proxy/0.log"
Oct 30 20:26:14.543356 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.543317    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/kube-rbac-proxy-rules/0.log"
Oct 30 20:26:14.666761 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:14.666689    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d49bbb759-7xwlv_62f34307-2c45-4d42-8340-8d4d2d63e43e/kube-rbac-proxy-metrics/0.log"
Oct 30 20:26:15.042353 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:15.042279    4320 scope.go:122] "RemoveContainer" containerID="7bc2cf330d2a2ffcdbdd513777ef48190267a5e9de2176abf533069356f0d801"
Oct 30 20:26:15.051026 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:15.051005    4320 scope.go:122] "RemoveContainer" containerID="974242dfa629e03148bc7513f77b1975d753bf9b70c937bcace9f680af7aec40"
Oct 30 20:26:15.058920 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:15.058898    4320 scope.go:122] "RemoveContainer" containerID="f55cb525122531501f60fc24cfc5c93bc51d6d12da2489b85bfd435b3d9bb034"
Oct 30 20:26:18.186114 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.186063    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-646cbbdc7f-wsw55_07648395-fb08-4c8c-a550-d0d0dfb0f657/console/0.log"
Oct 30 20:26:18.201441 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201406    4320 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"]
Oct 30 20:26:18.201871 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201851    4320 cpu_manager.go:414] "containerMap: removing container" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy"
Oct 30 20:26:18.201871 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201868    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy"
Oct 30 20:26:18.202050 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201894    4320 cpu_manager.go:414] "containerMap: removing container" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy"
Oct 30 20:26:18.202050 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201900    4320 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy"
Oct 30 20:26:18.202050 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201966    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd4fcad-85bf-4edb-b322-fbc4defad9ce" containerName="istio-proxy"
Oct 30 20:26:18.202050 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.201980    4320 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9d0999-76b8-45bd-87db-fd086b0ad274" containerName="istio-proxy"
Oct 30 20:26:18.207608 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.207576    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.223188 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.223162    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"]
Oct 30 20:26:18.322831 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.322802    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-proc\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.323017 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.322870    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-podres\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.323017 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.322896    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-sys\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.323160 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.323043    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm82p\" (UniqueName: \"kubernetes.io/projected/af0f73eb-374c-4002-8d04-f8ae9a4406a7-kube-api-access-jm82p\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.323160 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.323092    4320 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-lib-modules\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.328632 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.328593    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-529p7_6edeecf4-b1fb-42bf-8a2a-f0c379443841/download-server/0.log"
Oct 30 20:26:18.425554 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425528    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-lib-modules\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425708 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425577    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-proc\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425708 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425639    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-podres\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425708 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425660    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-sys\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425708 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425700    4320 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm82p\" (UniqueName: \"kubernetes.io/projected/af0f73eb-374c-4002-8d04-f8ae9a4406a7-kube-api-access-jm82p\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425707    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-lib-modules\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425746    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-proc\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425750    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-podres\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.425881 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.425780    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af0f73eb-374c-4002-8d04-f8ae9a4406a7-sys\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.445314 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.445217    4320 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm82p\" (UniqueName: \"kubernetes.io/projected/af0f73eb-374c-4002-8d04-f8ae9a4406a7-kube-api-access-jm82p\") pod \"perf-node-gather-daemonset-pbfcl\" (UID: \"af0f73eb-374c-4002-8d04-f8ae9a4406a7\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.525279 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.525250    4320 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:18.819226 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:18.819155    4320 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"]
Oct 30 20:26:18.823504 ip-10-0-130-217 kubenswrapper[4320]: W1030 20:26:18.823465    4320 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf0f73eb_374c_4002_8d04_f8ae9a4406a7.slice/crio-3495b986ad511f5d33c437f0dcc9a35b6ee526494f684a3de0799d1f311f30af WatchSource:0}: Error finding container 3495b986ad511f5d33c437f0dcc9a35b6ee526494f684a3de0799d1f311f30af: Status 404 returned error can't find the container with id 3495b986ad511f5d33c437f0dcc9a35b6ee526494f684a3de0799d1f311f30af
Oct 30 20:26:19.445839 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:19.445801    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl" event={"ID":"af0f73eb-374c-4002-8d04-f8ae9a4406a7","Type":"ContainerStarted","Data":"4b6728dc3d83ff9e2f73af5fc11ae874b40e2146dce7d011b49e0e61910a49e5"}
Oct 30 20:26:19.446276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:19.445861    4320 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl" event={"ID":"af0f73eb-374c-4002-8d04-f8ae9a4406a7","Type":"ContainerStarted","Data":"3495b986ad511f5d33c437f0dcc9a35b6ee526494f684a3de0799d1f311f30af"}
Oct 30 20:26:19.446276 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:19.445978    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:19.485223 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:19.485176    4320 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl" podStartSLOduration=1.485162924 podStartE2EDuration="1.485162924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:19.483753211 +0000 UTC m=+4927.519265672" watchObservedRunningTime="2026-10-30 20:26:19.485162924 +0000 UTC m=+4927.520675388"
Oct 30 20:26:21.920720 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:21.920681    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2nn6_b81a300d-dfb4-4853-b354-e541e6c0aec4/dns/0.log"
Oct 30 20:26:22.040494 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:22.040463    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2nn6_b81a300d-dfb4-4853-b354-e541e6c0aec4/kube-rbac-proxy/0.log"
Oct 30 20:26:22.163339 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:22.163317    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ckfxz_2357170c-bfb3-4ee9-925f-e84d351a2c86/dns-node-resolver/0.log"
Oct 30 20:26:23.496208 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:23.496177    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77469d6cc6-zjsth_401dd476-ba95-4b49-9e3c-3c7ad7b386c4/registry/0.log"
Oct 30 20:26:23.640500 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:23.640469    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wmz8_b56513e8-bfbe-4662-a549-6aaf87a53316/node-ca/0.log"
Oct 30 20:26:25.462461 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:25.462428    4320 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-pbfcl"
Oct 30 20:26:25.712303 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:25.712278    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7556cf8c574v98c_7f28ddd5-8e1c-44aa-bd4e-c78bec9644ca/istio-proxy/0.log"
Oct 30 20:26:28.654841 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:26:28.654807    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:26:31.831642 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:31.831589    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-j8c67_cede89a0-f33e-4db3-bcac-24ce856b418f/istio-proxy/0.log"
Oct 30 20:26:34.750143 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:34.750114    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-89vxf_59938fcc-4816-4ad1-9a47-413063935cb6/serve-healthcheck-canary/0.log"
Oct 30 20:26:36.426485 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:36.426456    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvd84_a87615c5-a046-4c52-8d19-df38cbb27340/kube-rbac-proxy/0.log"
Oct 30 20:26:36.541231 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:36.541203    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvd84_a87615c5-a046-4c52-8d19-df38cbb27340/exporter/0.log"
Oct 30 20:26:36.656246 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:36.656220    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvd84_a87615c5-a046-4c52-8d19-df38cbb27340/extractor/0.log"
Oct 30 20:26:40.342357 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:40.342323    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lhwvl_3ec07eb2-2af5-4d2b-ae2d-d905c57085f8/manager/1.log"
Oct 30 20:26:40.343288 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:40.343265    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lhwvl_3ec07eb2-2af5-4d2b-ae2d-d905c57085f8/manager/2.log"
Oct 30 20:26:40.512356 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:40.512316    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-7dx5m_61f6a874-9cc9-4ceb-a304-dab54cf6a8be/manager/0.log"
Oct 30 20:26:41.451095 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:41.451057    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-c757k_6ab50bb9-d775-4460-bcff-e18d445dc2a3/manager/0.log"
Oct 30 20:26:41.761664 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:41.761574    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-m6f5w_92948160-ac67-4192-ad60-83336fc1158f/server/0.log"
Oct 30 20:26:42.116008 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:42.115935    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gqrwf_41afb888-9cc2-486a-b5f7-5e6fa8863e06/manager/0.log"
Oct 30 20:26:42.590189 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:42.590153    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/manager/0.log"
Oct 30 20:26:42.718287 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:42.718265    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-8jpmq_85d88a98-d8df-4e5f-a0f1-c210590f6dad/copy-manifests/0.log"
Oct 30 20:26:45.386712 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:45.386680    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c54b56ff8-qzdjf_379c0868-30fe-4f71-be41-4bbaf3faea2e/manager/0.log"
Oct 30 20:26:57.690155 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:57.690127    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/kube-multus-additional-cni-plugins/0.log"
Oct 30 20:26:57.814710 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:57.814675    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/egress-router-binary-copy/0.log"
Oct 30 20:26:57.939852 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:57.939826    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/cni-plugins/0.log"
Oct 30 20:26:58.064479 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:58.064426    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/bond-cni-plugin/0.log"
Oct 30 20:26:58.188915 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:58.188894    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/routeoverride-cni/0.log"
Oct 30 20:26:58.313110 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:58.313086    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/whereabouts-cni-bincopy/0.log"
Oct 30 20:26:58.435743 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:58.435716    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gq2v8_c0885c92-5132-4361-901c-4530f23a5c57/whereabouts-cni/0.log"
Oct 30 20:26:58.646273 ip-10-0-130-217 kubenswrapper[4320]: E1030 20:26:58.646238    4320 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvd84" containerName="extractor"
Oct 30 20:26:59.729218 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:26:59.729185    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nppsv_c4b52d98-fb0b-4145-aaf5-dc3301172e4b/kube-multus/0.log"
Oct 30 20:27:00.386476 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:00.386441    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kn5zp_ca3c9c4d-c2a2-4fb7-a250-396d97523ad7/network-metrics-daemon/0.log"
Oct 30 20:27:00.499007 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:00.498972    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kn5zp_ca3c9c4d-c2a2-4fb7-a250-396d97523ad7/kube-rbac-proxy/0.log"
Oct 30 20:27:05.095380 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.095344    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/ovn-controller/0.log"
Oct 30 20:27:05.202637 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.202614    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/ovn-acl-logging/0.log"
Oct 30 20:27:05.329923 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.329896    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/kube-rbac-proxy-node/0.log"
Oct 30 20:27:05.452468 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.452447    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/kube-rbac-proxy-ovn-metrics/0.log"
Oct 30 20:27:05.574315 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.574292    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/northd/0.log"
Oct 30 20:27:05.698058 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.698031    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/nbdb/0.log"
Oct 30 20:27:05.822614 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:05.822543    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/sbdb/0.log"
Oct 30 20:27:06.098408 ip-10-0-130-217 kubenswrapper[4320]: I1030 20:27:06.098332    4320 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kgjxh_0aeb84c0-12d1-4c7b-9001-42a3f6b1391a/ovnkube-controller/0.log"